Methods, devices, systems and equipment for detecting violations of personal protective equipment

Abstract

This application discloses a method, apparatus, system, and equipment for detecting violations of personal protective equipment (PPE), relating to the field of image processing technology. The method includes: acquiring images of workers in an industrial setting and preprocessing these images; extracting complementary virtual multimodal features from the preprocessed images; inputting the virtual multimodal features into an improved target detection network; interacting and integrating the virtual multimodal features through a multimodal feature fusion mechanism; locating key detection points on the PPE based on the obtained fused features; and outputting the location information and recognition confidence of each key detection point; and performing decoupled detection and violation determination on each key detection point based on the location information and recognition confidence, thereby obtaining the violation detection result for each key detection point. This application can achieve accurate identification of subtle violations of PPE.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a method, device, system and equipment for detecting a rule violation state of individual protective equipment. BACKGROUND

[0002] In the field of industrial production safety management, the correct wearing of personal protective equipment (PPE) by workers is the key line of defense for protecting personal safety and avoiding production accidents. With the expansion of industrial scenarios and the complexity of production environments, the traditional PPE wearing detection mode relying on manual inspection has been difficult to meet the safety management needs. Under this background, the automatic PPE detection scheme based on computer vision and artificial intelligence technology has become the development trend of the industry. The core requirement is to process the real-time video stream collected by thousands of cameras in the factory through intelligent algorithms to accurately identify non-wearing of safety helmets, damage of protective clothing, and non-compliance of protective trousers, and thus improve the safety inspection efficiency and coverage, and eliminate the time and space blind area of manual inspection.

[0003] Currently, the mainstream automatic PPE detection technology mainly focuses on traditional feature extraction algorithms and early convolutional neural networks (CNN). Among them, the method based on manual feature extraction captures the shape and texture information of PPE by manually designing a feature template, but this method has poor adaptability to complex environments. In the case of light changes, background interference or local PPE occlusion, the feature expression ability is significantly reduced, and it is difficult to distinguish material details and subtle rule violation states. Although the image recognition algorithm based on early CNN improves the feature learning ability through deep learning, it still has the problems of high computational complexity and insufficient real-time performance. Moreover, the model performance is highly dependent on manual experience adjustment, and the scalability is poor when facing different industrial scene (such as mechanical processing workshop, chemical plant area) PPE detection needs, which requires a large amount of sample training and parameter optimization. In addition, the existing technology relies on a single RGB image modality for detection, and does not fully consider the complementarity of multi-source information in industrial scenarios. When facing complex working conditions such as night low light, personnel dense occlusion, etc., single modality data cannot provide enough discriminative information, resulting in a significant reduction in detection accuracy, which makes it difficult to meet the high-precision and high-real-time detection requirements of PPE wearing compliance in large-scale industrial scenarios.

[0004] To solve the limitations of single-modal detection, some existing technologies attempt to introduce multi-modal feature fusion ideas, such as a feature vector average fusion-based method and a feature fusion algorithm based on deep reinforcement learning. The former assumes that each modal feature element is equally important to the fusion result, and integrates the features through linear operations. Although the operation is simple and the calculation cost is low, it is easy to dilute high-discriminative features with low-quality features (such as clothing texture features disturbed by skeletal motion noise), and it is strict in the consistency of feature distribution and scale. Cross-modal data need to be normalized, otherwise the fusion effect will drop sharply. The latter is inspired by GraphSAGE inductive graph representation learning, and dynamically adjusts the fusion strategy through deep reinforcement learning. Although it can optimize feature representation, it needs a large amount of interactive data to train the strategy network, which takes a long time and costs a lot. Moreover, the model is strongly dependent on the graph topology structure. When the topology relationship changes dynamically due to the shielding of PPE key nodes (such as hats, sleeves, and cuffs), the network needs to be retrained, and the generalization ability is limited. Therefore, the above multi-modal fusion schemes have not formed an efficient fusion mechanism suitable for PPE key node (such as hat, clothing, and trousers) detection, and cannot achieve accurate identification of PPE subtle violation states. SUMMARY

[0005] Therefore, the present application provides a method, device, system and equipment for detecting the violation state of personal protective equipment, which can accurately identify the subtle violation state of PPE.

[0006] According to a first aspect of the present application, a method for detecting the violation state of personal protective equipment is provided, comprising:

[0007] Collecting an image of an operator in an industrial scene, and preprocessing the image of the operator, wherein the image of the operator contains the complete key area of the personal protective equipment worn by the operator;

[0008] Extracting complementary virtual multi-modal features from the preprocessed image of the operator, wherein the virtual multi-modal features are multiple complementary features with different information representation functions generated from the preprocessed image of the operator by a specific algorithm;

[0009] Inputting the virtual multi-modal features into an improved target detection network, interacting and integrating the virtual multi-modal features through a multi-modal feature fusion mechanism, locating each key detection point on the personal protective equipment based on the obtained fusion features, and outputting the position information and recognition confidence of each key detection point;

[0010] Based on the position information and recognition confidence of each key detection point, decoupling detection and violation judgment are performed on each key detection point respectively to obtain the violation state detection result of each key detection point.

[0011] According to a second aspect of the present application, a device for detecting a violation state of individual protective equipment is provided, comprising:

[0012] a processing module configured to collect an image of a worker in an industrial scene and pre-process the image of the worker, the image of the worker including a complete key area of individual protective equipment worn by the worker;

[0013] an extraction module configured to extract complementary virtual multi-modal features from the pre-processed image of the worker, the virtual multi-modal features being a plurality of complementary features of different information representation functions generated from the pre-processed image of the worker by a specific algorithm;

[0014] an input module configured to input the virtual multi-modal features into an improved target detection network, interact and integrate the virtual multi-modal features through a multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion features, and output position information and recognition confidence of the each key detection point;

[0015] a detection module configured to respectively perform decoupling detection and violation determination on the each key detection point based on the position information and recognition confidence of the each key detection point, and obtain a violation state detection result of the each key detection point.

[0016] According to a third aspect of the present application, a system for detecting a violation state of individual protective equipment is provided, comprising: an industrial network camera, a front-end network switch, a monitoring server, and a production management platform;

[0017] The industrial network camera is equipped with an electric zoom lens and is fixed to a wall or a load-bearing column through an L-shaped aluminum alloy bracket, and is configured to collect a video image stream of an industrial scene at a preset angle of view, the video image stream including a complete key area of individual protective equipment worn by a worker;

[0018] The front-end network switch is configured to transmit the video image stream to the monitoring server;

[0019] The monitoring server is configured to obtain a violation state detection result of each key detection point by executing the method for detecting a violation state of individual protective equipment according to any one of claims 1 to 7 based on the video image stream, and push violation information to the production management platform when a violation event trigger is detected;

[0020] The production management platform is configured to pop up an alarm pop-up window on a display and synchronously display a violation location map and real-time video.

[0021] According to a fourth aspect of the present application, a storage medium is provided, which stores a computer program, and the program, when executed by a processor, implements the above-mentioned individual protective equipment violation state detection method.

[0022] According to a fifth aspect of the present application, an electronic device is provided, which includes a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and the processor, when executing the program, implements the above-mentioned individual protective equipment violation state detection method.

[0023] Through the above technical solutions, the individual protective equipment violation state detection method, device, system and equipment provided by the present application can ensure that the core information required for determining equipment compliance is covered in the image by collecting industrial scene worker images containing the complete key area of the individual protective equipment worn by the worker and preprocessing the images, avoiding the detection blind spot caused by the missing of the key area, and improving the image quality through preprocessing to provide reliable basic input for the subsequent detection link. The complementary virtual multi-modal features are extracted from the preprocessed worker images, which can break through the limitation of insufficient information in the existing single RGB image mode in complex industrial environments (such as light changes and background interference), form a more comprehensive information coverage through multiple complementary features, enhance the adaptability to different complex scenes, and improve the anti-interference ability and discrimination at the feature level. The virtual multi-modal features are input into the improved target detection network, the features are interacted and integrated through the multi-modal feature fusion mechanism, and each key detection point on the individual protective equipment is located based on the fused features and the position information and recognition confidence are output, which can optimize the shortcomings of the existing target detection technology in multi-information integration and key part positioning, strengthen the feature representation ability through effective fusion of multi-modal features, improve the accuracy of key detection point positioning, and provide reliable data support for subsequent violation judgment. Based on the position information and recognition confidence of each key detection point, decoupling detection and violation judgment are performed respectively, which can avoid the problem of insufficient recognition of subtle violation state or false alarm and missed alarm caused by the rough determination method in the existing technology, accurately recognize the violation state of the individual protective equipment through targeted decoupling judgment, and meet the demand for high-precision detection of individual protective equipment wearing compliance in large-scale industrial scenes.

[0024] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0026] Figure 1 A system network connection schematic diagram of a system for detecting a violation state of individual protective equipment is shown;

[0027] Figure 2 A flowchart of a method for detecting a violation state of individual protective equipment is shown;

[0028] Figure 3 A flowchart of a method for detecting a violation state of individual protective equipment is shown;

[0029] Figure 4 A structure schematic diagram of a device for detecting a violation state of individual protective equipment is shown;

[0030] Figure 5 A structure schematic diagram of a device for detecting a violation state of individual protective equipment is shown;

[0031] In the drawings:

[0032] 110-industrial network camera, 1101-L type aluminum alloy bracket, 120-front-end network switch, 130-monitoring server, 140-production management platform. DETAILED DESCRIPTION

[0033] The application will be described in detail below with reference to the drawings and in conjunction with embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0034] Currently, the mainstream automatic PPE detection technology mainly focuses on traditional feature extraction algorithms and early convolutional neural networks (CNN). The method based on manual feature extraction captures the shape and texture information of PPE by manually designing a feature template. However, this method has poor adaptability to complex environments, and the feature expression ability significantly decreases in scenes with changes in lighting, background interference, or PPE partial occlusion, making it difficult to distinguish material details and subtle violations. Although the early CNN-based image recognition algorithm improves the feature learning ability through deep learning, it still has high computational complexity and insufficient real-time performance. Moreover, the model performance is highly dependent on manual experience adjustment, and the scalability is poor when facing different industrial scene (such as mechanical processing workshop, chemical plant area) PPE detection requirements, which requires a large amount of sample training and parameter optimization. In addition, existing technologies rely on single RGB image modalities for detection and do not fully consider the complementarity of multi-source information in industrial scenes. When faced with complex working conditions such as night low light, personnel dense occlusion, etc., single modal data cannot provide enough discriminative information, resulting in a significant reduction in detection accuracy, making it difficult to meet the high-precision and high-real-time detection requirements of PPE wearing compliance in large-scale industrial scenes.

[0035] To solve the limitations of single modal detection, some existing technologies attempt to introduce multi-modal feature fusion ideas, such as feature vector average fusion-based methods and deep reinforcement learning-based feature fusion algorithms. The former assumes that each modal feature element is equally important to the fusion result, and integrates the features through linear operations. Although it is simple to operate and has low computational cost, it is easy to dilute high-discriminative features with low-quality features (such as clothing texture features disturbed by skeletal motion noise), and it requires strict consistency in feature distribution and scale. Cross-modal data needs to be normalized, otherwise the fusion effect will drop sharply. The latter is inspired by GraphSAGE inductive graph representation learning, which dynamically adjusts the fusion strategy through deep reinforcement learning. Although it can optimize feature representation, it requires a large amount of interactive data to train the strategy network, which takes a long time and is costly. Moreover, the model is highly dependent on the graph topology, and when the topology relationship of PPE key nodes (such as hat belts and sleeves) changes dynamically due to occlusion, the network needs to be retrained, limiting the generalization ability. Therefore, the above multi-modal fusion schemes have not formed an efficient fusion mechanism that adapts to PPE key node (such as hat, clothing, and trousers) detection, and cannot achieve accurate identification of subtle violations of PPE.

[0036] To solve the above technical problems, an individual protective equipment violation state detection system is provided, as shown in Figure 1 The system includes an industrial network camera 110, a front-end network switch 120, a monitoring server 130, and a production management platform 140.

[0037] The industrial network camera 110 is equipped with a 3-9mm motorized zoom lens to adapt to different coverage ranges, is fixed to a wall surface or a load-bearing column through an L-shaped aluminum alloy bracket 1101, and is used to collect a video image stream of an industrial scene at a preset downward angle (such as 15-20° horizontally downward), wherein the video image stream contains the complete key area of the individual protective equipment worn by the worker; the front-end network switch 120 bears a data transmission function, can be connected to the industrial network camera 110 through a twisted shield six category network cable, and is used to transmit the video image stream to the monitoring server 130; the monitoring server 130 can execute a full-process detection method such as virtual multi-modal feature extraction, improved YOLOv8 network fusion positioning, and key detection point violation judgment, and accurately outputs the violation state of each key detection point. Specifically, based on the video image stream, by executing the individual protective equipment violation state detection method provided in the present application, the violation state detection result of each key detection point can be obtained, and when a violation event trigger is detected, the violation information can be pushed to the production management platform 140; the production management platform 140 is used to pop up an alarm pop-up window on a display and synchronously display a violation location map and real-time video recording. The complete key area of the individual protective equipment refers to the key parts including a safety helmet (head coverage area) worn by the worker, a protective clothing (torso and sleeve / neck opening area), and protective trousers (lower limbs and trouser leg area), so that the key detection points such as the hat top, hat band fixing point, neck opening closing point, sleeve opening closing point, trouser waist position point, and trouser leg coverage point can be accurately positioned subsequently.

[0038] Correspondingly, in order to fully describe the implementation manner of the embodiment, the embodiment further provides an individual protective equipment violation state detection method. Figure 1 The method can be applied to the individual protective equipment violation state detection system Figure 2 as described above, and specifically can be applied to the monitoring server 130. As shown in the method includes:

[0039] Step 210, collecting a worker image in an industrial scene, and pre-processing the worker image, wherein the worker image contains the complete key area of the individual protective equipment worn by the worker.

[0040] The industrial scene refers to a specific environment centered on industrial production, including mechanical processing workshops, chemical production plant areas, assembly workshops, and other places where operating personnel carry out production operations. Such scenes usually have production activities such as mechanical operation and material handling, and have clear requirements for the wearing compliance of individual protective equipment. The operating personnel image refers to a visual image data that clearly presents the shape of the operating personnel and the protective equipment they wear, and is the core visual basis for subsequent judgment of the wearing state of the protective equipment. The individual protective equipment refers to protective supplies worn by operating personnel to resist safety risks (such as impact, splashing, dust, etc.) in industrial scenes. Common types include safety helmets, protective clothing, protective trousers, protective shoes, etc., and are key equipment for protecting the personal safety of operating personnel. The complete key area of individual protective equipment refers to the core part of individual protective equipment for judging whether the wearing is compliant, such as the top and belt area of the safety helmet, the collar and sleeve area of the protective clothing, and the waist and leg area of the protective trousers. The completeness of these areas directly determines whether the subsequent protective equipment wearing can be accurately determined to comply with safety standards.

[0041] Collecting operating personnel images in industrial scenes can ensure the adaptability of data to actual application scenarios and avoid interference of non-industrial scene images on the detection model. Preprocessing the images can remove factors that affect detection accuracy such as blur and redundancy, improving image quality. Requiring the inclusion of complete key areas of individual protective equipment can avoid the problem of being unable to determine whether the equipment is worn in compliance due to the absence of key parts.

[0042] Step 220, extracting complementary virtual multi-modal features from the preprocessed operating personnel image.

[0043] The virtual multi-modal features refer to multiple complementary features of different information representation functions generated from a single preprocessed operating personnel image through specific algorithms (such as light decomposition, motion analysis, texture enhancement, etc.) without relying on multi-source original data collected by multiple real sensors (such as RGB cameras, infrared sensors, motion sensors, etc.).

[0044] Extracting complementary virtual multi-modal features from the preprocessed operating personnel image can break through the limitations of single-dimensional information in a single image and capture key information (such as shape, texture, and anti-interference properties) of individual protective equipment from different angles through multiple complementary features, effectively addressing the problem of insufficient discrimination ability of single features in complex scenarios. The complementarity of these features can provide more comprehensive and multi-dimensional information support for subsequent target detection networks, helping the network to more accurately locate the key detection points of individual protective equipment and reduce misjudgment or omission caused by one-sided information, thereby improving the accuracy and robustness of individual protective equipment violation state detection.

[0045] Step 230, input the virtual multi-modal feature into the improved target detection network, interact and integrate the virtual multi-modal feature through the multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion feature, and output the position information and recognition confidence of each key detection point.

[0046] Among them, the improved target detection network refers to the network model optimized and upgraded on the basis of the traditional target detection network (such as YOLO, Faster R-CNN, etc.) for the needs of multi-modal feature processing, key detection point positioning, etc. Usually, modules suitable for multi-modal fusion and structures for improving positioning accuracy are added to better meet the needs of individual protective equipment key detection point positioning; the multi-modal feature fusion mechanism refers to a special mechanism for processing multiple different types of features. The core function is to make the features of different modalities interact (such as strengthening the effective information related to protection equipment and weakening the environmental interference information) and integrate as a whole (such as converting features of different dimensions into unified dimension fusion features), so as to fully play the information complementary advantage of multi-modal features; the recognition confidence refers to the numerical value (usually the value range is 0-1) output by the target detection network for quantifying the reliability of the key detection point positioning result. The closer the value is to 1, the more reliable the positioning result of the key detection point is. The lower the value, the more likely there is deviation in the positioning result (such as being affected by shielding and interference), which provides a reference for the reliability of the subsequent violation judgment.

[0047] For the embodiments of the present disclosure, the virtual multi-modal features extracted in the early stage and having information complementarity can be input into the target detection network optimized and upgraded; then, through the special multi-modal feature fusion mechanism, the different types of virtual multi-modal features interact with each other (such as mutual reinforcement of effective information and supplement of missing information) and integrate as a whole (such as forming a unified feature representation); finally, based on the fusion feature obtained after integration, the specific positions of each key detection point on the individual protective equipment for judging the compliance of wearing are accurately determined, and the reliability (i.e. recognition confidence) of each key detection point positioning result is output, which provides direct data support for subsequent violation judgment.

[0048] The improved target detection network can further optimize feature processing and positioning accuracy, and the combination of the two can make the fusion features more comprehensive and accurate in reflecting the properties of individual protective equipment; positioning key detection points based on the fusion features can not only improve the accuracy of detection point location recognition, but also quantify the reliability of the positioning results through the output recognition confidence, effectively reducing the positioning deviation of key detection points caused by one-sided feature information or insufficient network performance, laying a solid foundation for subsequent accurate judgment of individual protective equipment violation status, and improving the accuracy and reliability of the detection process as a whole.

[0049] Step 240, based on the position information and recognition confidence of each key detection point, decoupling detection and violation judgment are performed on each key detection point respectively to obtain the violation status detection results of each key detection point.

[0050] Among them, the position information of the key detection point refers to the specific position data of each key detection point of the individual protective equipment in the image or space, which may be coordinate values in the image coordinate system, or distances / angles relative to reference points (such as the vertical distance from the top of the hat to the shoulder), and is the core data for judging whether the detection point meets the compliance position standard; decoupling detection refers to the independent detection of each key detection point, that is, the analysis of a detection point does not depend on the results of other detection points, nor is it bound to the judgment process of other detection points (such as analyzing the safety hat belt detection point, without considering the protective clothing sleeve detection point), ensuring that the analysis of each detection point is not disturbed by other factors; violation judgment refers to the process of comparing the position information of the key detection point with the preset compliance standard, and judging whether the detection point meets the safety specification combined with the recognition confidence; the violation status detection result refers to the final judgment conclusion output for each key detection point, which is usually divided into compliance and violation, and some results will also be labeled with the violation type (such as "position deviation" and "key part missing").

[0051] Decoupling detection and violation judgment based on the position information and recognition confidence of the key detection points can not only avoid the interference between different detection points (such as the judgment result of the safety hat detection point does not affect the judgment of the protective clothing detection point), ensuring that the analysis of each detection point is more focused and accurate, but also through the dual basis of position information and recognition confidence, considering the accuracy and reliability of the judgment, effectively reducing the misjudgment or omission caused by single basis judgment, and finally outputting accurate and independent violation status results for each key detection point, providing solid subdivision basis for the overall protective equipment compliance judgment.

[0052] In summary, the method for detecting the illegal state of personal protective equipment provided by the present application can ensure that the core information required for determining the compliance of equipment is covered in the image by collecting the industrial scene worker image containing the complete key area of the personal protective equipment worn by the worker and preprocessing the image, avoid the detection blind area caused by the missing of the key area, and improve the image quality through preprocessing to provide reliable basic input for the subsequent detection link. The complementary virtual multi-modal features are extracted from the preprocessed worker image, which can break through the limitation of insufficient information in the existing single RGB image mode under complex industrial environments (such as light changes and background interference), form a more comprehensive information coverage through multi-class complementary features, enhance the adaptability to different complex scenes, and improve the anti-interference ability and discrimination at the feature level. The virtual multi-modal features are input into the improved target detection network, the features are interacted and integrated by means of the multi-modal feature fusion mechanism, and each key detection point on the personal protective equipment is located based on the fused features and the position information and recognition confidence are output, which can optimize the shortcomings of the existing target detection technology in multi-information integration and key part positioning, strengthen the feature representation ability through effective fusion of multi-modal features, improve the accuracy of key detection point positioning, and provide reliable data support for subsequent illegal judgment. The decoupling detection and illegal judgment are performed based on the position information and recognition confidence of each key detection point, which can avoid the problem of insufficient recognition of subtle illegal state or false alarm and missing caused by the rough determination method in the prior art. Through targeted decoupling judgment, the illegal state of personal protective equipment can be accurately recognized to meet the demand for high-precision detection of the wearing compliance of personal protective equipment in large-scale industrial scenes.

[0053] Further, as a refinement and expansion of the above embodiment, in order to fully describe the implementation of the present embodiment, the present embodiment further provides another method for detecting the illegal state of personal protective equipment, as shown in Figure 3 The method comprises the following steps:

[0054] Step 310: Collecting the worker image in the industrial scene, and preprocessing the worker image, wherein the worker image contains the complete key area of the personal protective equipment worn by the worker.

[0055] For the present disclosure embodiment, the embodiment step 310 can specifically include the following steps:

[0056] Step 310-1: Collecting the video image stream of the industrial scene by using the industrial network camera, wherein the industrial network camera is deployed in the activity area of the worker, and is used to obtain the video image stream containing the complete key area at a preset angle of view.

[0057] The activity area of the worker is a specific area in which the worker carries out daily production-related activities (such as device operation, material transfer, process operation, etc.) in the industrial scene, is the focus coverage range of the video image stream collection, and ensures that the wearing state of the protective equipment of the worker can be captured. The video image stream is dynamic visual data composed of a series of time sequence image frames generated by continuous collection of the industrial network camera, and compared with a single static image, can more comprehensively reflect the action change of the worker and the dynamic state of the protective equipment. The preset angle of view is a shooting angle determined by pre-adjustment when the camera is installed, the lens is directed downward horizontally, the image composition can be optimized through the angle, the obstruction of obstacles in the scene to the key area of the protective equipment is reduced, and meanwhile, the overall state of the worker and the protective equipment can be ensured to be presented completely. The complete key area is a set of core parts directly related to the wearing compliance judgment of the individual protective equipment, for example, the top and belt area of the safety hat, the collar and sleeve area of the protective clothing, and the waist and leg area of the protective trousers, and the completeness of these areas is a prerequisite for subsequent detection.

[0058] For the embodiments of the present disclosure, an industrial network camera can be used as a collection device, which is installed in the activity area in which the worker carries out daily production operation (such as operating a device and carrying materials), and a shooting angle of the camera is preset. Through this deployment and angle setting, it is ensured that the continuous video image stream collected by the camera can completely present the key area of the individual protective equipment worn by the worker, avoids missing of the key area due to improper collection position or angle deviation, and provides original visual data meeting the detection requirements for subsequent image preprocessing, feature extraction and other links.

[0059] Step 310-2: Extracting image frames in the video image stream according to preset rules, and performing preliminary validity screening on the extracted image frames to eliminate invalid image frames in which the personnel contour is incomplete due to lens obstruction or image blur.

[0060] The preset rule is an extraction logic defined in advance before image frame extraction, and common forms can include “fixed time interval extraction” (such as extracting 2 frames per second), “event triggered extraction” (such as extracting additional frames when the worker enters a key work station), etc. The core purpose is to control the data volume to adapt to the computing resources while ensuring the representativeness of the data, and to balance the detection efficiency and effect.

[0061] For the embodiments of the present disclosure, discrete static image frames can be extracted from the continuous video image stream collected in the industrial scene according to pre-set logical rules (such as fixed time interval, action trigger of workers, etc.), so as to avoid resource waste caused by processing full-quantity video frames; then preliminary quality and effectiveness judgment can be carried out on each extracted image frame, and whether there is a problem of lens being blocked by equipment or materials, or image definition being insufficient due to environmental dust or motion blur is mainly checked; if the image frame has the problem of incomplete worker outline (especially the key area of the individual protective equipment worn) and cannot be identified, it is defined as an invalid image frame and is removed, and finally the valid image frame with clear image, complete worker outline and identifiable key area of protective equipment is retained, which provides qualified visual data input for subsequent feature extraction, key detection point positioning and the like.

[0062] According to the pre-set rule, the image frame can be extracted, so as to balance the detection real-time performance and data effectiveness while ensuring that the data covers the activity state of the worker, and avoid the calculation resource redundancy caused by processing the full frame of the continuous video stream.

[0063] In step 310-3, a lightweight algorithm is used to identify a potential worker area in the retained valid image frame, and the image in the potential worker area is subjected to grayscale normalization processing and denoising processing.

[0064] The potential worker area refers to an image area identified by the lightweight algorithm from the valid image frame, in which the worker is probably present, and is usually marked in the form of a rectangular frame. The area contains the complete body shape of the worker and the individual protective equipment worn by the worker, and is the focus of subsequent image optimization processing, rather than the whole image range.

[0065] For the embodiments of the present disclosure, the lightweight algorithm with low calculation resource consumption and fast running speed can be used to identify and frame the area (i.e. the potential worker area) in which the worker is probably present from the valid image frame (without blocking, clear image) retained after preliminary screening, so as to avoid invalid processing of the background area (such as equipment, wall, material) without workers in the image; further, the optimization processing can be carried out only on the image in the identified potential worker area, and specifically, the grayscale normalization processing can be used to adjust the grayscale value of the image in the area to a uniform range, so as to eliminate the problem of uneven brightness under different lighting conditions; at the same time, the denoising processing can be used to remove the image speckles in the area caused by the industrial environment (such as dust, equipment reflection, electromagnetic interference), so as to finally obtain the image data with clear worker area and identifiable details, which provides high-quality focused area data for subsequent virtual multi-modal feature extraction and key detection point positioning.

[0066] The potential personnel area is identified by using a lightweight algorithm, which can greatly reduce the consumption of computing resources under the premise of ensuring the accuracy of area positioning, avoid the waste of efficiency caused by indiscriminate processing of the whole image, and reduce the interference of the background environment (such as equipment and materials) on subsequent detection by focusing on the personnel area.

[0067] Step 320, complementary virtual multi-modal features are extracted from the pre-processed worker image.

[0068] The virtual multi-modal features at least include illumination invariance features, motion enhancement features, and texture enhancement features.

[0069] For the embodiments of the present disclosure, the embodiment step 320 can specifically include the following steps:

[0070] Step 320-1, the RetinexNet algorithm is used to decompose the pre-processed worker image to obtain a reflection component reflecting the essential properties of the worker and the personal protective equipment, and an illumination component reflecting the environmental illumination; the local binary pattern texture feature is superimposed on the reflection component to generate the illumination invariance feature by enhancing the edge profile and surface detail information in the reflection component.

[0071] For the embodiments of the present disclosure, the RetinexNet algorithm can be used to decompose the pre-processed image into a reflection component and an illumination component, wherein the reflection component records the inherent properties (such as color, texture, and contour) of the worker and the personal protective equipment (such as safety helmet and protective clothing), which are not affected by the change of environmental light, and the illumination component only reflects the illumination condition (such as strong light, shadow, and uneven light) of the current scene; further, the local binary pattern texture feature can be introduced and superimposed on the reflection component, which further enhances the edge profile (such as the brim line of the safety helmet and the collar edge of the protective clothing) and the surface details (such as the stitching and reflective stripes of the protective clothing) in the reflection component, and finally generates the illumination invariance feature with the anti-interference characteristic of illumination, which ensures that the feature focuses only on the essential properties of the protective equipment and is not affected by the fluctuation of environmental illumination.

[0072] The reflection component and the illumination component are obtained by decomposing the image by the RetinexNet algorithm, which can effectively eliminate the interference of environmental illumination on the image, so that the subsequent feature extraction only focuses on the essential properties of the protective equipment, avoiding the distortion of the feature caused by uneven illumination in the industrial scene (such as the difference between bright and dark workshop lights and backlight); the local binary pattern texture feature is superimposed on the reflection component, which can specifically enhance the edge and detail information of the protective equipment, solve the problem of detail blur in the reflection component, and make the features of key parts such as the safety helmet band and the protective clothing collar more prominent.

[0073] Step 320-2, based on the Farneback dense optical flow algorithm, calculate the pixel motion vector between the pre-processed images of the workers in continuous multiple frames, record the action trajectory of the workers in the continuous frames according to the pixel motion vector, and generate a motion history image that can reflect the action trend; combine the time domain difference feature to supplement and optimize the action details of the motion history image, and extract the motion enhancement feature from the optimized motion history image for reflecting the position change of individual protective equipment.

[0074] Wherein, the pixel motion vector is a vector data calculated by the Farneback algorithm, describing the position change of a single pixel in the continuous two frames of images, containing two core information of motion direction and motion distance; the motion history image is a visualized image that presents the motion trajectory in continuous multiple frames of images in the form of gray value accumulation, usually using different gray values to represent the time sequence of motion occurrence (such as high gray value of recent motion area, low gray value of early motion area), which can intuitively reflect the action trend and motion range of the object (worker), rather than the instantaneous motion state of a single frame; the time domain difference feature is a feature extracted from the dynamic change area in the image by calculating the gray value difference of the corresponding pixels between the continuous two frames or multiple frames of images, which can make up for the fuzzy problem of the motion history image in detail representation, and make the action information more complete.

[0075] For the embodiments of the present disclosure, based on the Farneback dense optical flow algorithm, the pre-processed images of the workers in continuous multiple frames can be calculated to obtain the motion direction and distance of each pixel in the image between frames (i.e. pixel motion vector), realizing the complete capture of the motion state of all pixels in the image; then according to these pixel motion vectors, the overall action path of the workers in the continuous image frames (such as bending, lifting hands, turning around, etc.) can be tracked and recorded, and the trajectory can be accumulated in a visualized form to generate a motion history image, which can intuitively reflect the action trend of the workers, rather than the instantaneous state of a single frame; finally, the time domain difference feature (by calculating the pixel gray value change between adjacent frames) can be introduced to supplement and optimize the action details that may be blurred or missing in the motion history image (such as the slight displacement of the protective clothing sleeve when the worker lifts his hands), and finally the motion enhancement feature that can accurately reflect the position change of individual protective equipment (such as safety helmet, protective clothing, protective trousers) with the action of the workers can be extracted from the optimized motion history image.

[0076] The Farneback dense optical flow algorithm is used to calculate the pixel motion vector, which can cover the motion information of all pixels in the image, avoid the problem of incomplete motion trajectory caused by sparse optical flow only capturing local key points, and ensure the comprehensiveness of the motion trajectory record of the worker; the generated motion history image can convert the instantaneous action of the continuous frame into intuitive trend information, which is convenient for quickly identifying the action direction and amplitude of the worker; the action details are optimized by combining the time domain difference characteristics, which can make up for the deficiency of the motion history image in representing small actions, and make the subtle position changes of the protective equipment (such as the slight displacement of the safety helmet when bending over, and the offset of the sleeve when lifting the hand) more clear.

[0077] In step 320-3, the pre-processed worker image is filtered by using a Gabor filter set to obtain a texture response graph with enhanced detail texture; and a multi-scale direction gradient histogram is extracted from the texture response graph, and the multi-scale direction gradient histogram is converted into a texture enhancement feature by counting the gradient distribution characteristics in different scales and different directions.

[0078] For the embodiments of the present disclosure, a Gabor filter set composed of multiple different parameter (scale, direction) Gabor filters can be used to filter the pre-processed worker image. Specifically, the multi-scale characteristics of the filter can capture different sizes of textures in the image (such as wide textures of reflective strips on protective clothing and fine textures of stitches), and the multi-directional characteristics can cover textures in different directions (such as horizontal hat brim lines and vertical protective clothing zipper textures), and finally output a texture response graph that can significantly enhance the detail texture of the protective equipment, making the originally blurred or background disturbed texture information more prominent; then, a multi-scale direction gradient histogram can be extracted from the texture response graph, and the gradient distribution of image pixels in different scales (such as 1x1 and 3x3 pixel ranges) and different directions (such as 0°, 45°, 90°, etc.) can be counted (the gradient reflects the edge and direction of the texture), and the histogram data obtained by counting can be converted into a structured texture enhancement feature, which can accurately represent the texture properties of individual protective equipment and provide feature support at the detail level for subsequent detection.

[0079] The multi-scale and multi-directional characteristics of the Gabor filter set used for filtering can comprehensively cover different types of texture information of individual protective equipment, avoid missing key textures (such as not being able to capture vertical stitches with only horizontal filters), and effectively enhance the distinguishability of the detail texture of the protective equipment and the background; the multi-scale direction gradient histogram extracted from the texture response graph can convert the intuitive texture visual information into calculable structured data, and further enhance the feature recognition of the texture by counting the gradient distribution, making the features of subtle textures such as reflective strips on protective clothing, safety helmet strap buckles, and protective trousers seam lines more prominent.

[0080] Step 330, input the virtual multi-modal feature into the improved target detection network, interact and integrate the virtual multi-modal feature through the multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion feature, and output the position information and recognition confidence of each key detection point.

[0081] wherein the improved target detection network is an improved YOLOv8 network, the improved YOLOv8 network comprises a backbone network, an illumination intensity detection module, a feature fusion module reconstructed by using a depth separable convolution, and a feature aggregation layer, and a cross attention module is introduced at the end of the backbone network.

[0082] For the embodiments of the present disclosure, the embodiment step 330 can specifically include the following steps:

[0083] Step 330-1, determine the RGB original feature corresponding to the pre-processed worker image, input the virtual multi-modal feature and the RGB original feature into the backbone network, realize feature interaction through the cross attention module at the end of the backbone network, take the RGB original feature as the query vector, take the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature as the key value vector respectively, perform element-level fusion on the virtual multi-modal feature and the RGB original feature through attention weight calculation, and obtain intermediate interaction features with multi-modal anti-interference ability and basic visual clarity.

[0084] For the embodiments of the present disclosure, the RGB original feature corresponding to the pre-processed worker image can be determined first, and the dimension preprocessing is carried out on the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature (i.e. the virtual multi-modal feature) extracted in the early stage. Through interpolation, dimension reduction or dimension increase, the dimensions of the three types of virtual features are uniformly adjusted to be completely consistent with the RGB original feature, so as to eliminate the fusion obstacles caused by the dimension difference. Then, the three types of virtual multi-modal features (illumination invariance feature, motion enhancement feature, and texture enhancement feature) extracted in the early stage are input into the backbone network of the target detection network together with the RGB original feature. Then, the cross attention module at the end of the backbone network is relied on to realize the interaction between different features. Specifically, the RGB original feature can be taken as the query vector, and the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature can be taken as the key value vector respectively. By calculating the attention weight (quantifying the effective contribution degree of different virtual features) of the query vector and each key value vector, the weighted virtual multi-modal feature and the RGB original feature are element-level fused, and finally the intermediate interaction feature with multi-modal anti-interference ability and basic visual clarity is generated, which lays a double advantage foundation for subsequent feature integration and key detection point positioning.

[0085] Correspondingly, the method can specifically include the following steps: performing dimension preprocessing on the virtual multi-modal feature, and adjusting the virtual multi-modal feature to a feature dimension consistent with the RGB original feature; taking the RGB original feature as a query vector, and taking the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature as key-value vectors respectively, and calculating attention weights of the query vector and each key-value vector respectively; adjusting the attention weights of each key-value vector based on an effective information threshold; weighting and multiplying the adjusted attention weights and the corresponding virtual multi-modal features respectively to obtain each modal feature highlighting effective information, and then element-level adding the modal features and the key region feature of the RGB original feature to obtain an intermediate interaction feature after interference is weakened.

[0086] The RGB original feature is a red (R), green (G), and blue (B) three-channel visual feature directly extracted from the preprocessed worker image, and contains core visual information such as the basic color, overall contour, and spatial position of the protective equipment. The query vector is a feature vector used for active information retrieval in the cross-attention module, which is the RGB original feature in this scheme. The core role of the query vector is to determine the target of feature interaction (i.e., which supplementary information needs to be obtained from other features), and to ensure that the fused feature retains the basic visual clarity. The key-value vector is a feature vector used to provide supplementary information in the cross-attention module, which is the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature in this scheme. The key-value vector corresponds to anti-interference information of different dimensions, and supplements the required anti-interference capability by matching calculation with the query vector. The attention weight is a value (usually in the range of 0-1) quantifying the contribution of the key-value vector to the query vector in the cross-attention module. The higher the weight, the more effective information the key-value vector (such as the illumination invariance feature) has in the current scene, and the more its feature needs to be retained. The lower the weight, the lower the contribution, and the more its influence needs to be weakened. The effective information threshold is a judgment standard (such as 0.5) for screening the attention weight. The virtual feature with a weight higher than the threshold is determined to contain high-value effective information, and its weight is retained or strengthened. The virtual feature with a weight lower than the threshold is determined to have insufficient effective information, and its weight is suppressed or set to zero to filter interference. The intermediate interaction feature is a transitional feature generated after dimension preprocessing, weight calculation and adjustment, and weighted fusion, and has both the basic visual clarity of the RGB original feature and the anti-interference capability of the virtual multi-modal feature. It is the core input for subsequent feature integration.

[0087] The attention weight is calculated by taking the RGB original feature as a query and the virtual feature as a key value, so that the system can dynamically identify the anti-interference information most needed in the current scene (such as preferentially retaining the light invariance feature weight when the light is complex), and avoid feature redundancy caused by indiscriminate fusion; the weight is adjusted based on the effective information threshold, which can filter low-value interference information (such as weakening the low weight of the texture enhancement feature when the background is simple), and improve the feature purity; the combination operation of weighted multiplication and element-level addition highlights the anti-interference ability of the high-value virtual feature through the weight, and relies on the RGB key area feature to retain the basic visual details of the protective equipment, and the intermediate interaction feature generated by the deep integration of the two can effectively resist multiple types of interference such as light, motion, background, etc. in the industrial scene, while ensuring that the outline and key parts of the protective equipment are clear and identifiable, providing high-quality feature support for subsequent feature further integration and accurate positioning of key detection points, and significantly reducing the detection deviation caused by insufficient feature quality.

[0088] In step 330-2, the learnable weight parameters corresponding to the light invariance feature, the motion enhancement feature and the texture enhancement feature are set respectively, the average brightness value of the image of the current work scene is calculated through the light intensity detection module, and the parameter value of the learnable weight parameter is dynamically adjusted according to the average brightness value of the image.

[0089] For the embodiments of the present disclosure, the light invariance feature, the motion enhancement feature and the texture enhancement feature extracted in the early stage can be respectively configured with independent learnable weight parameters. Such parameters are not fixed values, and can be adjusted according to the scene demand in the model training or inference process, and are used to quantify the contribution priority of different features in the current scene; through a special light intensity detection module, the brightness of the image of the current work scene is calculated (such as calculating the average gray value of all pixels of the image), and the average brightness value of the image representing the brightness of the scene (the higher the value, the stronger the light, and the lower the value, the weaker the light) is obtained; further, the learnable weight parameters of the three types of features can be dynamically adjusted based on the average brightness value of the image, for example, the weight of the light invariance feature is increased when the light is weak, so that it can fully play the role of resisting interference in dark environment; when the light is normal, the weights of the three types of features are balanced, so that the detail capturing ability of the motion and texture features is not weakened, and finally the feature weight is accurately adapted to the scene light.

[0090] The learnable weight parameters are set for the three types of virtual features, which can break the limitation of traditional fixed weight that cannot adapt to scene changes, and allow the feature contribution to have a dynamic adjustment space; the scene brightness is obtained in real time through the light intensity detection module, which can provide an objective quantitative basis for weight adjustment and avoid the adaptation deviation caused by subjective setting; the weight is dynamically adjusted according to the brightness value, so that the feature most suitable for the current light scene (such as the light invariance feature in weak light) can obtain a higher priority, fully play its anti-interference or detail capturing ability, and at the same time avoid introducing invalid information due to the weight of other features being too high.

[0091] Step 330-3, the intermediate interaction features are integrated by using a feature fusion module to obtain intermediate fusion features.

[0092] The intermediate fusion features are the features output by the feature fusion module after integrating the intermediate interaction features, and the core feature is information focusing and no redundancy. The anti-interference ability and basic visual advantage of the intermediate interaction features are retained, and the repeated information is eliminated through integration, and the feature signals of the key detection points (such as the top of the safety helmet and the collar of the protective clothing) of the individual protective equipment are strengthened.

[0093] For the embodiments of the present disclosure, the intermediate interaction features generated through cross-attention interaction and weight adjustment in the early stage can be input into the feature fusion module. The module integrates the dispersed information (such as contour information resistant to light interference, displacement information capturing action, and detail information highlighting texture) in the intermediate interaction features through a preset feature processing logic (such as depth separable convolution, feature splicing and optimization, and key information strengthening). Through this operation, the redundant information that may exist between the features can be eliminated, and the effective features related to the key detection points of the individual protective equipment can be strengthened, and finally the intermediate fusion features with more focused information and more accurate feature representation are output, providing better feature input for subsequent feature aggregation and key detection point positioning.

[0094] The integration of the intermediate interaction features by using the feature fusion module can effectively solve the problems of dispersed multi-modal information and redundant information in the intermediate interaction features. Through the targeted processing of the module, the dispersed anti-interference information and basic visual information are converted into unified features focusing on the key areas of the individual protective equipment. At the same time, the integration process can further strengthen the feature recognition of the key detection points of the protective equipment (such as highlighting the feature signals of the hatband fixing point and the sleeve cuff joint point), and weaken the interference of irrelevant background or secondary information, so that the generated intermediate fusion features are more suitable for the needs of the subsequent target detection network for positioning the key detection points, laying a key foundation for improving the accuracy and recognition confidence of the position recognition of the key detection points, and avoiding the positioning deviation caused by the dispersion of feature information.

[0095] Step 330-4, the intermediate fusion features are processed by the feature aggregation layer to unify the feature dimensions and enhance the details, and the fusion features reflecting the appearance, motion state and texture details of the individual protective equipment are generated.

[0096] In a specific application scenario, since the intermediate fusion features can include sub-features from different modalities and different processing stages (e.g., some sub-features have a dimension of 128x128 and some have a dimension of 256x256), in this embodiment, all sub-features can be adjusted to a uniform dimension by interpolation, dimension mapping, and other techniques to eliminate subsequent detection network input obstacles caused by dimension differences. In addition, detail enhancement processing can also be performed simultaneously. By introducing edge detection, texture enhancement, and other sub-algorithms, the subtle key information of individual protective equipment (e.g., safety hat belt buckle, protective clothing sleeve seam, and protective trousers leg edge) in the intermediate fusion features can be enhanced to compensate for the possible weakening of the details in the early fusion process. Finally, through these two processes, fusion features that cover the appearance, dynamic state, and surface texture details of protective equipment are generated, achieving comprehensive and unified representation of multi-dimensional information.

[0097] In step 330-5, based on the fusion features, independent decoupled detection branches respectively set for each key detection point in the improved YOLOv8 network are used to generate feature heat maps corresponding to each key detection point, the coordinates of each key detection point are extracted from the feature heat maps as position information, and the recognition confidence of each key detection point is calculated.

[0098] For the embodiments of the present disclosure, the fusion features covering the appearance, motion state, and texture details of individual protective equipment processed by the feature aggregation layer in the early stage can be first input into the improved YOLOv8 network. The network has independently decoupled detection branches respectively set for different key detection points such as the safety hat top, hat belt fixing point, protective clothing collar closing point, and sleeve closing point. Each branch focuses only on the feature analysis of a single key detection point to avoid interference between the features of different detection points. Through the targeted processing of the fusion features by each independent branch, a feature heat map corresponding to each key detection point is generated (the higher the brightness of the area in the heat map, the higher the probability of the existence of the key detection point). Then, the pixel coordinates are extracted from the brightness peak area of the feature heat map as the position information of the key detection point. At the same time, the reliability of the coordinate positioning result is quantified by the probability calculation logic built in the branch, and the recognition confidence of each key detection point is output.

[0099] The independent decoupled detection branches of the improved YOLOv8 network can avoid cross interference between the features of different key detection points (e.g., the safety hat detection point feature does not affect the protective clothing detection point analysis), ensure the positioning focus of each detection point, and significantly improve the positioning accuracy of a single detection point. The feature heat map can convert abstract fusion features into intuitive brightness distribution, making the position of the key detection point more identifiable and reducing the error of coordinate extraction. The recognition confidence provides a quantitative reliability basis for the positioning result, which can avoid blind judgment of the detection points with ambiguous positioning (e.g., blocked).

[0100] Step 340, for different categories of key detection points, respectively based on the position information to calculate the violation judgment index.

[0101] The key detection points include at least a safety helmet key detection point, a protective clothing key detection point, and a protective trousers key detection point. The position information of the safety helmet key detection point includes at least a hat top position point and a hatband fixing point. The position information of the protective clothing key detection point includes at least a collar closure point and a sleeve opening fastening point. The protective trousers key detection point includes at least a waist position point and a trouser leg covering point.

[0102] For the safety helmet key detection point, the vertical distance between the hat top and the shoulder can be calculated by the coordinate difference between the hat top position point and the shoulder feature point. The visible pixel ratio of the hatband fixing point in the preset hatband area can be calculated by using the pixel ratio statistical method. For the protective clothing key detection point, the horizontal distance of the collar can be calculated by the coordinate difference between the left and right sides of the collar closure point. The pixel ratio of the sleeve opening fastening area to the total sleeve opening area can be calculated by analyzing the pixels in the sleeve opening fastening area. For the protective trousers key detection point, the vertical distance between the waist position point and the hip can be calculated by the coordinate difference between the waist position point and the hip feature point. The vertical distance between the bottom of the trouser leg and the ankle can be calculated by the coordinate difference between the trouser leg covering point and the ankle feature point.

[0103] Step 350, if the recognition confidence is greater than or equal to the preset confidence threshold, the violation judgment is performed according to the preset violation standard and the violation judgment index, and the violation state detection result of each key detection point is obtained.

[0104] The preset confidence threshold is a critical value (such as 0.8) for judging whether the positioning result of the key detection point has the qualification for judgment, which is the access threshold for starting the violation judgment. Only when the recognition confidence is greater than or equal to the threshold, the subsequent violation judgment link is entered, otherwise the judgment is temporarily not performed to avoid misjudgment. The preset violation standard is a key detection point compliance judgment standard (usually a quantitative range) based on industrial safety specifications or scene requirements, such as "the visible pixel ratio of the safety helmet hatband is greater than or equal to 80%", "the horizontal distance of the protective clothing collar is less than or equal to 3 cm", "the vertical distance between the protective trousers leg covering and the ankle is less than or equal to 2 cm", etc., which is the comparison basis of the violation judgment index. The violation judgment index is the quantitative data calculated based on the position information of the key detection point, which is used to reflect the actual state of the detection point, such as the vertical distance between the hat top and the shoulder, the visible pixel ratio of the hatband, the pixel ratio of the sleeve opening fastening area, etc., which is the core basis for comparison with the preset violation standard.

[0105] In a specific application scenario, only when the recognition confidence is greater than or equal to the preset confidence threshold, the position information (such as coordinates, pixel ratio, etc.) of the detection point is considered to be reliably positioned, and has the basis to carry out the violation judgment; if the confidence does not meet the standard, it is not directly judged to avoid misjudgment. When the recognition confidence is greater than or equal to the preset confidence threshold, the violation judgment index can be compared with the preset violation standard. If the index meets the standard, it is determined that the detection point is compliant, and if the index exceeds or does not meet the standard, it is determined to be in violation. Finally, the explicit violation state detection result is output for each key detection point.

[0106] Taking the recognition confidence threshold as the judgment premise can effectively filter out detection point data with unreliable positioning (such as low confidence), avoid misjudgment caused by position information deviation, and ensure the reliability of the judgment result from the source; by comparing the preset violation standard with the violation judgment index, subjective compliance judgment is converted into objective quantitative data comparison, eliminating the subjectivity and randomness of manual judgment, and ensuring the uniformity of the judgment standard for different key detection points.

[0107] Step 360, if the recognition confidence is less than the preset confidence threshold, a Kalman filtering algorithm is started, the current position is predicted based on the historical position trajectory of the key detection point in the previous frame, and the violation judgment index is recalculated in combination with the predicted position.

[0108] For the embodiments of the present disclosure, when the recognition confidence of the key detection point is lower than the preset confidence threshold, the Kalman filtering algorithm can be started, and the reasonable position of the detection point in the current frame can be predicted based on the historical position trajectory (such as the change of the coordinates of the previous 5 frames of cap band fixed points) of the key detection point in the previous continuous image frames in combination with the filtering model; then, the violation judgment index of the detection point is recalculated based on the predicted current position as the new position, ensuring that even if the positioning of the current frame is unreliable, compliance judgment can still be achieved through historical data supplement, avoiding missed judgment or misjudgment caused by single frame positioning deviation.

[0109] In a specific application scenario, after decoupling detection and violation judgment are performed on each key detection point based on the position information and recognition confidence of each key detection point, and the violation state detection result of each key detection point is obtained, as an optional way, the embodiment steps can further include: when the violation state detection result is that there is a violation, automatically generating violation information containing a violation timestamp, a violation type, and a scene screenshot, and pushing the violation information to a production management platform, popping up an alarm pop-up window on the display of the production management platform, and synchronously displaying a violation position map and a real-time video; and storing the violation information to a cloud server to form a violation database supporting time, area, and violation type retrieval.

[0110] When a violation is detected, automatically generate a violation message containing key information, which can avoid information omission or errors caused by manual recording, ensure the integrity and accuracy of the violation data; real-time push to the production management platform and pop-up alarm, synchronous display of location map and real-time video, can let the management personnel locate the violation scene and master the violation details at the first time, greatly shorten the response time of violation, and stop unsafe behavior in time; store the violation information to the cloud to form a searchable database, which can realize long-term retention of violation data and provide data support for safety management through multi-dimensional search function.

[0111] In summary, the technical scheme of the present application can accurately collect images of complete key areas of PPE through an industrial network camera and pre-process to ensure input quality, which can solve the problem of weak detection basis caused by missing or poor quality of image key information in traditional detection; by generating three types of complementary virtual multi-modal features of illumination invariance, motion enhancement and texture enhancement, the limitations of single RGB modality in distinguishing insufficient information under complex working conditions such as illumination change, motion blur and background interference can be broken through, and the anti-interference ability and detail capturing force of feature representation can be greatly improved; relying on the improved YOLOv8 network with cross-attention module, illumination intensity detection module and depth separable convolution fusion module, efficient interaction integration of multi-modal features and accurate positioning of key detection points are realized, which takes into account detection accuracy and real-time performance, avoiding the defects of traditional CNN such as complex calculation and poor scalability; then through the hierarchical violation judgment mechanism based on confidence threshold, subtle violation states such as hat belt not tightened and sleeve not buckled are accurately identified, reducing the false positive and false negative rate; finally, combined with the automatic generation, push pop-up window and cloud storage of violation information, the real-time and traceability of violation disposal are realized. The overall scheme effectively solves the pain points of existing PPE detection technology such as weak anti-interference ability, low positioning accuracy, rough judgment and inefficient management, meets the detection needs of high precision, high robustness and high real-time performance of PPE wearing compliance in large-scale industrial scenes, and provides efficient and reliable technical support for industrial production safety management.

[0112] Further, as a specific implementation of the method shown in Figure 2 and Figure 3 , the embodiment provides a violation state detection device of individual protective equipment, as shown in Figure 4 , the device comprises a processing module 41, an extraction module 42, an input module 43 and a detection module 44.

[0113] The processing module 41 can be used for collecting images of workers in industrial scenes and pre-processing the images of workers, wherein the images of workers contain complete key areas of individual protective equipment worn by workers;

[0114] The extraction module 42 can be used to extract complementary virtual multi-modal features from the pre-processed worker image, and the virtual multi-modal features are multiple complementary features with different information representation functions generated from the pre-processed worker image by a specific algorithm.

[0115] The input module 43 can be used to input the virtual multi-modal features into the improved target detection network, interact and integrate the virtual multi-modal features through a multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion features, and output the position information and recognition confidence of each key detection point.

[0116] The detection module 44 can be used to respectively perform decoupling detection and rule violation determination on each key detection point based on the position information and recognition confidence of each key detection point, and obtain the rule violation state detection result of each key detection point.

[0117] In some embodiments of the present application, the processing module 41 can be specifically used to collect a video image stream of an industrial scene by using an industrial network camera, the industrial network camera is deployed in an activity area of a worker, and is used to obtain a video image stream containing a complete key area at a preset angle of view; image frames in the video image stream are extracted according to a preset rule, the extracted image frames are preliminarily screened for validity, and invalid image frames in which a personnel contour is incomplete due to lens obstruction or image blur are removed; a lightweight algorithm is used to identify a potential personnel area in the retained valid image frames, and the image of the potential personnel area is subjected to grayscale normalization processing and denoising processing.

[0118] In some embodiments of the present application, the virtual multi-modal features at least include illumination invariance features, motion enhancement features and texture enhancement features; accordingly, the extraction module 42 can be specifically used to decompose the pre-processed worker image by using the RetinexNet algorithm to obtain the reflection component reflecting the essential properties of the worker and the individual protective equipment, and the illumination component reflecting the environmental illumination; superimpose the local binary pattern texture features to the reflection component, and generate the illumination invariance features by strengthening the edge profile and surface detail information in the reflection component; calculate the pixel motion vector between the continuous multiple frames of pre-processed worker images based on the Farneback dense optical flow algorithm, record the action trajectory of the worker in the continuous frames according to the pixel motion vector, and generate the motion history image which can reflect the action trend; supplement and optimize the action details of the motion history image in combination with the time domain difference features, extract the motion enhancement features from the optimized motion history image for reflecting the position change of the individual protective equipment; filter the pre-processed worker image by using the Gabor filter bank to obtain the texture response map with enhanced detail texture; extract the multi-scale orientation histogram of gradient from the texture response map, and convert the multi-scale orientation histogram of gradient into the texture enhancement features by counting the gradient distribution features in different scales and different directions.

[0119] In some embodiments of the present application, the improved target detection network is an improved YOLOv8 network, the improved YOLOv8 network includes a backbone network, an illumination intensity detection module, a feature fusion module reconstructed by using a depth separable convolution, and a feature aggregation layer, and a cross-attention module is introduced at the end of the backbone network; the input module 43 can be specifically used to determine the RGB original features corresponding to the preprocessed worker image, input the virtual multi-modal features and the RGB original features into the backbone network, realize feature interaction through the cross-attention module at the end of the backbone network, take the RGB original features as a query vector, take the illumination invariance features, the motion enhancement features, and the texture enhancement features as key-value vectors respectively, perform element-level fusion on the virtual multi-modal features and the RGB original features through attention weight calculation, and obtain intermediate interaction features with multi-modal anti-interference ability and basic visual clarity; the learnable weight parameters corresponding to the illumination invariance features, the motion enhancement features, and the texture enhancement features are set respectively, the image average brightness value of the current work scene is calculated through the illumination intensity detection module, and the parameter values of the learnable weight parameters are dynamically adjusted according to the image average brightness value; the intermediate interaction features are integrated through the feature fusion module to obtain intermediate fusion features; the feature dimensions of the intermediate fusion features are unified and the details are enhanced through the feature aggregation layer to generate fusion features that comprehensively reflect the appearance, motion state and texture details of individual protective equipment; based on the fusion features, through the independent decoupling detection branch respectively set in the improved YOLOv8 network for each key detection point, the feature heat map corresponding to each key detection point is generated, the coordinates of each key detection point are extracted from the feature heat map as position information, and the recognition confidence of each key detection point is calculated.

[0120] In some embodiments of the present application, when the virtual multi-modal features and the RGB original features are input into the backbone network, the feature interaction is realized through the cross-attention module at the end of the backbone network, the RGB original features are taken as a query vector, the illumination invariance features, the motion enhancement features, and the texture enhancement features are taken as key-value vectors respectively, the element-level fusion is performed on the virtual multi-modal features and the RGB original features through attention weight calculation, and the intermediate interaction features with multi-modal anti-interference ability and basic visual clarity are obtained, the input module 43 can be specifically used to perform dimension preprocessing on the virtual multi-modal features, and adjust the virtual multi-modal features to the same feature dimension as the RGB original features; the RGB original features are taken as a query vector, the illumination invariance features, the motion enhancement features, and the texture enhancement features are taken as key-value vectors respectively, and the attention weights of the query vector and each key-value vector are calculated respectively; the attention weights of each key-value vector are adjusted based on an effective information threshold; the adjusted attention weights are multiplied with the corresponding virtual multi-modal features respectively to obtain each modal feature highlighting effective information, and then the element-level addition is performed on the key region features of the RGB original features to obtain the intermediate interaction features after interference is weakened.

[0121] In some embodiments of the present application, the detection module 44 can be specifically used to calculate a violation judgment index based on the position information for different categories of key detection points; if the recognition confidence is greater than or equal to a preset confidence threshold, a violation judgment is performed according to a preset violation standard and the violation judgment index to obtain a violation state detection result of each key detection point; if the recognition confidence is less than the preset confidence threshold, a Kalman filtering algorithm is started to predict the current position based on the historical position trajectory of the key detection point corresponding to the previous frame, and the violation judgment index is recalculated in combination with the predicted position.

[0122] In some embodiments of the present application, as shown in Figure 5 , the device further comprises a pushing module 45 and a storage module 46.

[0123] The pushing module 45 can be used to automatically generate violation information containing a violation timestamp, a violation type, and a field screenshot when the violation state detection result is that there is a violation, and push the violation information to a production management platform, pop up an alarm pop-up window on the display of the production management platform, and synchronously display a violation position map and real-time video; and,

[0124] The storage module 46 can be used to store the violation information to a cloud server to form a violation database supporting time, area, and violation type retrieval.

[0125] It should be noted that other corresponding descriptions of the functions of the individual protective equipment violation state detection device provided in the present embodiment can be referred to the corresponding descriptions in Figure 2 and Figure 3 , which will not be described here.

[0126] Based on the above methods as shown in Figure 2 and Figure 3 , correspondingly, the present embodiment also provides a storage medium having a computer program stored thereon, which is executed by a processor to implement the individual protective equipment violation state detection method as shown in Figure 2 and Figure 3 .

[0127] Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions to make an electronic device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various implementation scenarios of the present application.

[0128] Based on the above methods as shown in Figure 2 and Figure 3 , and Figure 4 andFigure 5 In order to achieve the above-mentioned purposes, the virtual device embodiment shown, the embodiments of the present application also provide an electronic device, which can be a personal computer, a tablet computer, a server, or other network devices, etc., the device comprising a storage medium and a processor; the storage medium is used for storing a computer program; the processor is used for executing the computer program to realize the above-mentioned Figure 2 And Figure 3 The method for detecting the violation state of the individual protective equipment.

[0129] Optionally, the above-mentioned entity device can also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The optional user interface can also include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0130] Those skilled in the art can understand that the above-mentioned entity device structure provided by the embodiments does not constitute a limitation on the entity device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0131] The storage medium can also include an operating system and a network communication module. The operating system is a program for managing the hardware and software resources of the above-mentioned entity device, supporting the running of information processing programs and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing entity device.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software plus necessary general hardware platforms, or by hardware.

[0133] The embodiment of the application can solve the problem of weak detection basis caused by missing or poor quality of image key information in traditional detection by accurately collecting images of PPE complete key areas through an industrial network camera and ensuring input quality through preprocessing; by generating three types of complementary virtual multi-modal features of illumination invariance, motion enhancement and texture enhancement, the limitations of insufficient discrimination information of single RGB mode under complex working conditions such as illumination change, motion blur and background interference can be broken through, and the anti-interference ability and detail capturing ability of feature representation can be greatly improved; relying on the improved YOLOv8 network with cross attention module, illumination intensity detection module and depth separable convolution fusion module, efficient interaction integration of multi-modal features and accurate positioning of key detection points are realized, the detection accuracy and real-time performance are considered, and the defects of traditional CNN such as complex calculation and poor expansibility are avoided; then through the hierarchical violation judgment mechanism based on confidence threshold, subtle violation states such as hat belt not tightened and sleeve not buckled are accurately identified, and the false positive and false negative rates are reduced; finally, combined with the automatic generation, push pop-up window and cloud storage of violation information, the real-time and traceability of violation disposal are realized. The overall scheme effectively solves the pain points of weak anti-interference ability, low positioning accuracy, rough judgment and inefficient management of existing PPE detection technology, meets the detection requirements of high precision, high robustness and high real-time performance of PPE wearing compliance in large-scale industrial scenes, and provides efficient and reliable technical support for industrial production safety management.

[0134] Those skilled in the art can understand that the modules or processes in the drawings are not necessarily required for the implementation of the present application. Those skilled in the art can understand that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the description of the implementation scenario, or can be changed and located in one or more devices different from the implementation scenario. The modules of the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0135] The above application serial numbers are only for description, and do not represent the advantages and disadvantages of the implementation scenario. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present application.

Claims

1. A method of detecting a violation state of personal protective equipment, characterized by, The method comprises the following steps: Collecting an image of an operator in an industrial scene and preprocessing the image of the operator, wherein the image of the operator contains a complete key area of individual protective equipment worn by the operator; Extracting complementary virtual multi-modal features from the preprocessed image of the operator, wherein the virtual multi-modal features are multiple complementary features with different information representation functions generated from the preprocessed image of the operator by a specific algorithm, and the virtual multi-modal features at least include illumination invariance features, motion enhancement features and texture enhancement features; Inputting the virtual multi-modal features into an improved target detection network, interacting and integrating the virtual multi-modal features through a multi-modal feature fusion mechanism, locating each key detection point on the individual protective equipment based on the obtained fusion features, and outputting the position information and recognition confidence of each key detection point, wherein the improved target detection network is an improved YOLOv8 network, and the improved YOLOv8 network comprises a backbone network, an illumination intensity detection module, a feature fusion module reconstructed by a depth separable convolution, and a feature aggregation layer, and a cross-attention module is introduced at the end of the backbone network; Inputting the virtual multi-modal features into an improved target detection network, interacting and integrating the virtual multi-modal features through a multi-modal feature fusion mechanism, locating each key detection point on the individual protective equipment based on the obtained fusion features, and outputting the position information and recognition confidence of each key detection point, comprising: determining the corresponding RGB original features of the preprocessed image of the operator, inputting the virtual multi-modal features and the RGB original features into the backbone network, realizing feature interaction through the cross-attention module at the end of the backbone network, taking the RGB original features as query vectors, taking the illumination invariance features, the motion enhancement features and the texture enhancement features as key value vectors, calculating the attention weight, element-level fusing the virtual multi-modal features and the RGB original features to obtain intermediate interaction features with multi-modal anti-interference ability and basic visual clarity; setting the learnable weight parameters corresponding to the illumination invariance features, the motion enhancement features and the texture enhancement features, calculating the average image brightness value of the current work scene through the illumination intensity detection module, and dynamically adjusting the parameter values of the learnable weight parameters according to the average image brightness value; integrating the intermediate interaction features through the feature fusion module to obtain intermediate fusion features; performing feature dimension unification and detail enhancement processing on the intermediate fusion features through the feature aggregation layer to generate fusion features that comprehensively reflect the appearance, motion state and texture details of the individual protective equipment; based on the fusion features, generating feature heat maps corresponding to each key detection point through independent decoupling detection branches respectively set in the improved YOLOv8 network for each key detection point, extracting the coordinates of each key detection point as position information from the feature heat maps, and calculating the recognition confidence of each key detection point. Based on the position information and the recognition confidence of each key detection point, decoupling detection and violation judgment are performed on each key detection point respectively to obtain a violation state detection result of each key detection point.

2. The method of claim 1, wherein, The image of the worker in the industrial scene is collected, and the image of the worker is preprocessed, including: An industrial network camera is used to collect a video image stream of an industrial scene, the industrial network camera is deployed in an activity area of a worker, and is used to obtain a video image stream containing the complete key area at a preset angle of view; Image frames in the video image stream are extracted according to a preset rule, and the extracted image frames are subjected to preliminary validity screening, and invalid image frames in which a personnel contour is incomplete due to lens obstruction or image blur are removed; A lightweight algorithm is used to identify a potential personnel region in the retained valid image frames, and the image of the potential personnel region is subjected to grayscale normalization processing and denoising processing.

3. The method of claim 1, wherein, Complementary virtual multi-modal features are extracted from the preprocessed image of the worker, including: A RetinexNet algorithm is used to decompose the preprocessed image of the worker to obtain a reflection component reflecting the essential properties of the worker and individual protective equipment, and an illumination component reflecting the environmental illumination; a local binary pattern texture feature is superimposed on the reflection component to generate an illumination-invariant feature by enhancing the edge contour and surface detail information in the reflection component; Based on a Farneback dense optical flow algorithm, pixel motion vectors between consecutive multiple preprocessed images of the worker are calculated, the motion trajectory of the worker in consecutive frames is recorded according to the pixel motion vectors, and a motion history image reflecting the motion trend is generated; the motion details of the motion history image are supplemented and optimized in combination with a time domain difference feature, and a motion enhancement feature reflecting the position change of the individual protective equipment is extracted from the optimized motion history image; A Gabor filter bank is used to filter the preprocessed image of the worker to obtain a texture response map with enhanced detail texture; a multi-scale orientation histogram of gradient is extracted from the texture response map, and the multi-scale orientation histogram of gradient is converted into a texture enhancement feature by counting the gradient distribution characteristics in different scales and different directions.

4. The method of claim 1, wherein, The virtual multi-modal features and the RGB original features are input into the backbone network, feature interaction is realized through a cross-attention module at the end of the backbone network, the RGB original features are used as a query vector, the illumination-invariant feature, the motion enhancement feature and the texture enhancement feature are used as key-value vectors, element-level fusion of the virtual multi-modal features and the RGB original features is performed through attention weight calculation, intermediate interaction features with multi-modal anti-interference capability and basic visual clarity are obtained, including: The virtual multi-modal features are subjected to dimension preprocessing, and the virtual multi-modal features are adjusted to the same feature dimension as the RGB original features. Taking the RGB original feature as a query vector, taking the illumination invariance feature, the motion enhancement feature, and the texture enhancement feature as key value vectors respectively, attention weights of the query vector and each key value vector are calculated respectively; Based on the effective information threshold, the attention weights of each key value vector are adjusted; The adjusted attention weights are multiplied with the corresponding virtual multi-modal features respectively to obtain multi-modal features highlighting effective information, and then the multi-modal features are added with the key region features of the RGB original feature at the element level to obtain intermediate interaction features after interference is weakened.

5. The method of claim 1, wherein, Based on the position information and the recognition confidence of each key detection point, decoupling detection and violation judgment are performed on each key detection point respectively to obtain the violation state detection result of each key detection point, including: For key detection points of different categories, a violation judgment index is calculated based on the position information; If the recognition confidence is greater than or equal to a preset confidence threshold, violation judgment is performed according to a preset violation standard and the violation judgment index to obtain the violation state detection result of each key detection point; If the recognition confidence is less than the preset confidence threshold, a Kalman filtering algorithm is started, the current position is predicted based on the historical position trajectory of the key detection point in the previous frame, and the violation judgment index is recalculated in combination with the predicted position.

6. The method of claim 1, wherein, After the position information and the recognition confidence of each key detection point are used to perform decoupling detection and violation judgment on each key detection point to obtain the violation state detection result of each key detection point, the method further includes: When the violation state detection result is that there is a violation, violation information containing a violation timestamp, a violation type, and a field screenshot is automatically generated, the violation information is pushed to a production management platform, an alarm pop-up window is popped up on a display of the production management platform, a violation position map and real-time video are synchronously displayed; and The violation information is stored in a cloud server to form a violation database supporting time, area, and violation type retrieval.

7. A device for detecting a violation state of personal protective equipment, characterized by, The method comprises: a processing module configured to collect an image of an operator in an industrial scene and pre-process the image of the operator, the image of the operator containing a complete key region of individual protective equipment worn by the operator; an extraction module configured to extract complementary virtual multi-modal features from the pre-processed image of the operator, the virtual multi-modal features being a plurality of complementary features of different information representation functions generated from the pre-processed image of the operator by a specific algorithm, the virtual multi-modal features at least including an illumination invariance feature, a motion enhancement feature, and a texture enhancement feature; The input module is used to input the virtual multi-modal feature into the improved target detection network, interact and integrate the virtual multi-modal feature through a multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion feature, and output the position information and recognition confidence of each key detection point. The improved target detection network is an improved YOLOv8 network, and the improved YOLOv8 network includes a backbone network, an illumination intensity detection module, a feature fusion module reconstructed by using a depth separable convolution, and a feature aggregation layer. The cross attention module is introduced at the end of the backbone network. The input module is used to input the virtual multi-modal feature into the improved target detection network, interact and integrate the virtual multi-modal feature through a multi-modal feature fusion mechanism, locate each key detection point on the individual protective equipment based on the obtained fusion feature, and output the position information and recognition confidence of each key detection point. The improved target detection network is an improved YOLOv8 network, and the improved YOLOv8 network includes a backbone network, an illumination intensity detection module, a feature fusion module reconstructed by using a depth separable convolution, and a feature aggregation layer. The cross attention module is introduced at the end of the backbone network. The detection module is used to decouple and detect each key detection point based on the position information and recognition confidence of each key detection point, and determine the violation state of each key detection point to obtain the violation state detection result of each key detection point.

8. A system for detecting a violation state of personal protective equipment, the system comprising: It includes an industrial network camera, a front-end network switch, a monitoring server, and a production management platform. ​ The industrial network camera is equipped with an electric zoom lens and is fixed to a wall surface or a load-bearing column through an L-shaped aluminum alloy bracket, is used for collecting a video image stream of an industrial scene at a preset low-angle shooting angle, and the video image stream contains complete key areas of individual protective equipment worn by workers; The front-end network switch is used for transmitting the video image stream to the monitoring server; The monitoring server is used for obtaining violation state detection results of each key detection point by executing the individual protective equipment violation state detection method in any one of claims 1 to 6 based on the video image stream, and pushing violation information to the production management platform when a violation event trigger is detected; The production management platform is used for popping up an alarm pop-up window on a display and synchronously displaying a violation location map and real-time video recording.

9. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 6.

Images