Intelligent safety tool access detection method based on AI vision

Abstract

The invention provides an intelligent safety tool access detection method based on AI vision. The method comprises the steps that 1, a tool access state data set in a tool cabinet is manufactured; 2, an attention mechanism-based improved YOLOv7s network is constructed as a tool access state detection model; 3, the data set is divided into a training set, a verification set and a test set according to a certain proportion, and training, verification and performance testing are carried out on the tool access state detection model; and step 4, using the deployed tool state detection model to carry out tool real-time access detection. The method is based on an artificial intelligence visual technology, rapid and accurate automatic identification, classification, access and monitoring are carried out on the tools, manual operation errors can be effectively reduced, the access process is accelerated, the safety tool management accuracy and efficiency can be improved, the safety tool loss risk is reduced, the safety tool registration and return process is simplified, and the safety tool management efficiency is improved. And management negligence is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of tool-assisted management, and in particular to an intelligent safety tool access detection method based on AI vision. Background Art

[0002] Safe tool management in a hydropower plant is crucial to ensuring the safe operation of the entire hydropower plant. Effective tool management can not only prevent the safety of workers and equipment caused by missing tools or inadequate supervision, but also improve the work efficiency of workers and reduce delays caused by waiting or repairing tools.

[0003] At present, the safety tool cabinets in power plants usually use ordinary safety tool cabinets for storage and access, or use intelligent cabinets that integrate RFID (radio frequency identification) and gravity sensing technology. Although these technologies have improved management efficiency to a certain extent and solved the problem of delayed information update, they have specific requirements for the use environment and have limitations in recognition accuracy, which restricts their widespread application.

[0004] Ordinary storage and retrieval methods of safety tool cabinets do not have an efficient storage and retrieval design, resulting in the staff being unable to trace the time, person, and tool used after storing and retrieving the tools, resulting in a lack of effective management.

[0005] Although RFID technology can record the access of safety tools in real time, it has certain requirements for the use environment. Strong magnetic interference may affect normal use. There are challenges in the application of RFID in certain environments (such as metal, liquid, etc.). For example, electronic tags such as screwdrivers and wrenches are not easy to embed into objects. When identifying multiple targets, the bit error rate may affect system performance.

[0006] In addition, existing storage and access can also use intelligent safety tool cabinets with gravity sensing technology. Although it can monitor the storage and access of safety tools in real time, it is less used in the management of safety tools in power plants. The main reason is that the gravity sensing system needs regular calibration and maintenance to ensure its accuracy and reliability, which may bring additional workload and cost. In addition, if the gravity sensing system is not accurate enough, false alarms may occur. Gravity sensing technology may be insensitive to certain types or shapes of tools, resulting in the inability to accurately detect the use of these tools. Summary of the invention

[0007] The present invention provides an AI vision-based intelligent safety tool access detection method, aiming to solve the above-mentioned technical problems.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: An AI vision-based intelligent safety tool access detection method, comprising: Step 1, create a data set of tool access status in the tool cabinet; Step 2. Build an improved YOLOv7s network based on the attention mechanism as a tool access status detection model, that is, integrate the hybrid attention mechanism into the SPPCSCP module of the classic YOLOv7s neck structure, and name the module SPPCBAM; Step 3: Divide the data set into training set, validation set and test set according to a certain ratio, and train, validate and perform performance testing on the tool access status detection model; Step 4. Save the tool access status detection model and deploy the model, and use the deployed tool status detection model to perform real-time tool access detection.

[0009] The above-mentioned Step 1 specifically includes the following steps: Step 1.1, collect monitoring video data of each layer of the tool cabinet: install cameras on each layer of the tool cabinet to achieve full coverage monitoring of each layer of the cabinet and obtain real-time monitoring data; Step 1.2, video frame processing: Use the VideoCapture function in the OpenCV extension library cv2 to perform frame processing on the collected monitoring video data of each layer of the tool cabinet to obtain the original image data set with a resolution of 1280 pixels × 720 pixels; Step 1.3, Dataset annotation: Use the image calibration tool Labelimg to manually calibrate each tool in the image and obtain a standard XML format file. All images and XML format files together constitute the tool access detection dataset.

[0010] In the above Step 2, the network framework of the improved YOLOv7s network based on the attention mechanism is: The improved YOLOv7 model consists of input, image initialization feature extraction layer, backbone network feature extraction module, neck structure feature fusion module and head structure target detection module. The image initialization feature extraction layer is used to preprocess the image and scale the image to a set size.

[0011] The backbone network feature extraction module mentioned above is used to extract shallow graphic features and deep semantic features of the image, and transmit three feature maps of different scales to the neck structure. The backbone network is equipped with CBS-2 module, CBS-3 module, ELAN module and MP-1 module. The ELAN module is a residual structure whose basic modules are CBS-1 and CBS-2, and the MP-1 module is a dual-branch downsampling structure whose basic modules are CBS-1 and CBS-3. The backbone network feature extraction module is used to extract shallow graphic features and deep semantic features of the image, and obtain feature maps of different scales.

[0012] The neck structure feature fusion module is provided with a CBS-1 module, a SPPCBAM module, an ELAN-W module, and an MP-2 module. The neck structure feature fusion module has an upsampling branch and a downsampling branch to achieve a better fusion of shallow graphic features and deep semantic features, and obtain feature maps of different scales. The ELAN-W module is a residual structure, and the basic modules are CBS-1 and CBS-2; The MP-2 module is a dual-branch downsampling structure, and the basic modules are CBS-1 and CBS-3.

[0013] The above head structure consists of three branches, corresponding to three detection layers of different scales. Each detection layer is responsible for predicting the category and location information of the object. Each branch consists of a REP module and a CBM module.

[0014] The SPPCBAM module in the above neck structure is a spatial pyramid pooling operation that integrates a hybrid attention mechanism, as follows: The input F of the spatial pyramid pooling operation with integrated hybrid attention mechanism is divided into sub-regions of three sizes, generating feature maps F1 with sub-regions of 5×5, F2 with sub-regions of 9×9, and F3 with sub-regions of 13×13 respectively; Perform global average pooling and global maximum pooling on the sub-region of F1 to obtain two feature vectors; pass these two feature vectors through a shared multi-layer perceptron MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F1 channel by channel to generate an enhanced feature map F1'; perform average pooling and maximum pooling on the channel-enhanced feature map F1' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F1' element by element to generate the final enhanced feature map F1"; Perform global average pooling and global maximum pooling on the sub-region of F2 to obtain two feature vectors; pass these two feature vectors through the shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F2 channel by channel to generate an enhanced feature map F2'; perform average pooling and maximum pooling on the channel-enhanced feature map F2' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F2' element by element to generate the final enhanced feature map F2"; Perform global average pooling and global maximum pooling on the sub-region of F3 to obtain two feature vectors; pass these two feature vectors through a shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F3 channel by channel to generate an enhanced feature map F3'; perform average pooling and maximum pooling on the channel-enhanced feature map F3' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F3' element by element to generate the final enhanced feature map F3".

[0015] In Step 3 above, the tool access status detection model is trained, verified, and tested as follows: Step 3.1. Set the hyperparameters required for training, including epoch, batch_size, learning rate, training optimizer, and loss function. The loss function is as follows: ; In the formula, , , represents the loss weight, represents the bounding box loss calculated using the IoU loss, represents the confidence loss calculated using binary cross entropy, represents the target detection loss calculated using binary cross entropy; Step 3.2, set model evaluation indicators: select 5 evaluation indicators commonly used in target detection tasks, namely intersection over union ratio, accuracy, precision, recall rate and F1-score; Step 3.3, initialize network parameters; Step 3.4, input the training set into the constructed improved YOLOv7s network, forward propagate the training data in the improved YOLOv7s network, and calculate the loss according to the network output. According to the calculated loss, back propagate and update the network parameters through the gradient descent algorithm until the training is completed; Step 3.5, freeze the network parameters, input the verification set into the trained improved YOLOv7s network, and optimize and adjust the model through evaluation indicators; Step 3.6, input the test set into the optimized improved YOLOv7s network to evaluate the performance of the network.

[0016] The forward propagation process in the above training process is as follows: Use the image initialization feature extraction layer to perform preliminary feature extraction on the input image to obtain an initial feature map; The backbone network feature extraction module of the improved YOLOv7s network is used, including the CBS-2 module of the first layer of the backbone network feature extraction module, the CBS-3 module of the second layer of the backbone network feature extraction module, the CBS-2 module of the third layer of the backbone network feature extraction module, the CBS-3 module of the fourth layer of the backbone network feature extraction module, the ELAN module of the fifth layer of the backbone network feature extraction module, the MP-1 module of the sixth layer of the backbone network feature extraction module, the ELAN module of the seventh layer of the backbone network feature extraction module, the MP-1 module of the eighth layer of the backbone network feature extraction module, the ELAN module of the ninth layer of the backbone network feature extraction module, the MP-1 module of the tenth layer of the backbone network feature extraction module, and the ELAN module of the eleventh layer of the backbone network feature extraction module, to perform image feature extraction from shallow to deep on the initial feature map.

[0017] The neck structure feature extraction module using the improved YOLOv7s network includes the SPPCBAM module of the first layer of the sampling branch of the neck structure feature fusion module, the CBS-1 module of the second layer of the sampling branch of the neck structure feature fusion module, the Upsample module of the third layer of the sampling branch of the neck structure feature fusion module, the Concat module of the fourth layer of the sampling branch of the neck structure feature fusion module, the ELAN-W module of the fifth layer of the sampling branch of the neck structure feature fusion module, the CBS-1 module of the sixth layer of the sampling branch of the neck structure feature fusion module, the Upsample module of the seventh layer of the sampling branch of the neck structure feature fusion module, and the The Concat module of the eighth layer of the upsampling branch of the structural feature fusion module, the ELAN-W module of the ninth layer of the upsampling branch of the neck structural feature fusion module, the MP-2 module of the first layer of the downsampling branch of the neck structural feature fusion module, the Concat module of the second layer of the downsampling branch of the neck structural feature fusion module, the ELAN-W module of the third layer of the downsampling branch of the neck structural feature fusion module, the MP-2 module of the fourth layer of the downsampling branch of the neck structural feature fusion module, the Concat module of the fifth layer of the downsampling branch of the neck structural feature fusion module, and the ELAN-W module of the sixth layer of the downsampling branch of the neck structural feature fusion module achieve better fusion of features of different scales; The Concat module of the fourth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN module feature map of the ninth layer of the backbone network feature extraction module; the Concat module of the eighth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN module feature map of the seventh layer of the backbone network feature extraction module; the Concat module of the second layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN-W module feature map of the fifth layer of the sampling branch of the neck structure feature fusion module; the Concat module of the fifth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the SPPCBAM module feature map of the first layer of the sampling branch of the neck structure feature fusion module; Finally, the features obtained by the ELAN-W module of the ninth layer of the upsampling branch of the neck structure feature fusion module, the ELAN-W module of the third layer of the downsampling branch of the neck network feature fusion module, and the ELAN-W module of the sixth layer of the downsampling branch of the neck network feature fusion module are used as inputs of three different scales of the head structure target detection module in the improved YOLOv7, and three corresponding decoding layers, namely a REP module and a CBM module, are used for decoding to obtain the candidate box position information, target category and confidence.

[0018] The above Step 4 specifically includes the following steps: Step 4.1, save the model framework and model parameters of the tool access status detection model when the evaluation index is optimal during the verification process; Step 4.2, deploy the tool access status detection model in the AI ​​vision-based intelligent safety tool access detection system; Step 4.3, whenever the cabinet door is closed, the tool access status detection model is linked to realize dynamic detection of the tool access status.

[0019] The present invention provides an intelligent safety tool access detection method based on AI vision. Compared with the prior art, the present invention has the following technical effects: 1) Based on artificial intelligence vision technology, the present invention designs an intelligent safety tool access detection method based on AI vision, which realizes online and accurate monitoring of the access status of each tool in the tool cabinet, eliminates the safety tool access registration or RFID scanning registration process, effectively improves the access efficiency; at the same time, it reduces the risk of loss of safety tools.

[0020] 2) The present invention designs an improved YOLOv7s network based on the attention mechanism as a security tool access status detection model, which enables the network to focus on the extraction of key information from two dimensions, channel and space, during the image feature extraction process, while ignoring the extraction of irrelevant information, thereby improving the detection accuracy.

[0021] 3) The present invention installs cameras on each layer of tools and tools, and links the safety tool access status detection model when the tool cabinet door is closed to detect the safety tools in the tool cabinet. Compared with the visual method in which the camera is installed on the upper end of the cabinet and the safety tool access detection is realized through access actions, the detection accuracy of the present invention is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a flow chart of the present invention; Figure 2 This is a schematic diagram of the improved YOLOv7s network in the present invention; Figure 3 It is a schematic diagram of the SPPCBAM structure in the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following content will systematically and completely describe the specific technical solutions of the present invention in combination with the drawings provided according to the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] like Figure 1-Figure 3 As shown in, an AI vision-based intelligent safety tool access detection method includes: Step 1, creating a data set of tool access status in a tool cabinet, including the following steps: Step 1.1, collect monitoring video data of each layer of the tool cabinet: install cameras on each layer of the tool cabinet to achieve full coverage monitoring of each layer of the cabinet and obtain real-time monitoring data; Step 1.2, video frame processing: Use the VideoCapture function in the OpenCV extension library cv2 to perform frame processing on the collected monitoring video data of each layer of the tool cabinet, and obtain 10,000 original image data sets with a resolution of 1280 pixels × 720 pixels; Step 1.3, data set annotation: Use the image calibration tool Labelimg to manually calibrate each tool in the image and obtain a standard XML format file. All images and XML format files together constitute the tool access detection data set.

[0025] Step 2: construct an improved YOLOv7s network based on the attention mechanism as a tool access status detection model, that is, integrate the hybrid attention mechanism into the SPPCSCP module of the classic YOLOv7s neck structure, and name the module SPPCBAM; Figure 2 As shown; The improved YOLOv7s consists of input, image initialization feature extraction layer, backbone network feature extraction module, neck structure feature fusion module and head structure target detection module.

[0026] The image initialization feature extraction layer is mainly used to preprocess the input image (monitoring images of each layer of the tool cabinet). The Padding function is used to adjust the input image to a square size, and then it is scaled proportionally to an image with a size of 640 pixels × 640 pixels and a depth of 3.

[0027] The backbone network feature extraction module is used to extract shallow graphic features and deep semantic features of the image, and transmit three feature maps of 20×20×1024, 40×40×1024, and 80×80×512 to the neck structure. The backbone network is mainly composed of CBS-2 module, CBS-3 module, ELAN module and MP-1 module. Among them, CBS-2 is a convolution layer with a stride of 1 and a convolution kernel size of 3×3, which is mainly used for feature extraction. CBS-2 is a convolution layer with a stride of 2 and a convolution kernel size of 3×3, which is mainly used for downsampling. The ELAN module is a residual structure with CBS-1 and CBS-2 as the basic modules. The MP-1 module is a dual-branch downsampling structure with CBS-1 and CBS-3 as the basic modules. Among them, CBS-1 is a convolution layer with a stride of 1 and a convolution kernel size of 1×1, which is mainly used to change the number of channels of the feature map.

[0028] The neck structure feature fusion module has an upsampling branch and a downsampling branch, which are used to achieve better fusion of shallow graphic features and deep semantic features, and pass three feature maps of 20×20×512, 40×40×256, and 80×80×128 to the head structure. The neck structure is mainly composed of CBS-1 module, SPPCBAM module, ELAN-W module, and MP-2 module. Among them, the ELAN-W module is a residual structure, and the basic modules are CBS-1 and CBS-2. The MP-2 module is a dual-branch downsampling structure, and the basic modules are CBS-1 and CBS-3.

[0029] The head structure consists of three branches, corresponding to three detection layers of different scales. Each detection layer is responsible for predicting the category and location information of the object. Each branch consists of a REP module and a CBM module.

[0030] Preferably, the SPPCBAM module in the neck structure is a spatial pyramid pooling operation integrating a hybrid attention mechanism, as follows: The input F of the spatial pyramid pooling operation with integrated hybrid attention mechanism is divided into sub-regions of three sizes, generating feature maps F1 with sub-regions of 5×5, F2 with sub-regions of 9×9, and F3 with sub-regions of 13×13 respectively; Perform global average pooling and global maximum pooling on the sub-region of F1 to obtain two feature vectors; pass these two feature vectors through a shared MLP (Multilayer Perceptron) to generate a channel attention map; multiply the channel attention map with the sub-region of F1 channel by channel to generate an enhanced feature map F1'; perform average pooling and maximum pooling on the channel-enhanced feature map F1' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F1' element by element to generate the final enhanced feature map F1"; Perform global average pooling and global maximum pooling on the sub-region of F2 to obtain two feature vectors; pass these two feature vectors through the shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F2 channel by channel to generate an enhanced feature map F2'; perform average pooling and maximum pooling on the channel-enhanced feature map F2' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F2' element by element to generate the final enhanced feature map F2"; Perform global average pooling and global maximum pooling on the sub-region of F3 to obtain two feature vectors; pass these two feature vectors through a shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F3 channel by channel to generate an enhanced feature map F3'; perform average pooling and maximum pooling on the channel-enhanced feature map F3' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F3' element by element to generate the final enhanced feature map F3".

[0031] Step 3, dividing the data set into a training set, a validation set, and a test set in a ratio of 7:2:1, and training, validating, and performance testing the tool access status detection model, including the following steps: Step 3.1, set the hyperparameters required for training, including setting epoch=800, batch_size=64, learning rate=0.0001, training optimizer as Adam optimizer, and loss function; the loss function is expressed as follows: ; In the formula, , , represents the loss weight, represents the bounding box loss calculated using the IoU loss, represents the confidence loss calculated using binary cross entropy, represents the target detection loss calculated using binary cross entropy; Step 3.2, set model evaluation indicators: select five evaluation indicators commonly used in target detection tasks, namely intersection over union, accuracy, precision, recall and F1-score; Step 3.3, initialize network parameters; Step 3.4, input the training set into the constructed improved YOLOv7s network, forward propagate the training data in the improved YOLOv7s network, and calculate the loss according to the network output. According to the calculated loss, back propagate and update the network parameters through the gradient descent algorithm until the training is completed; Step 3.5, freeze the network parameters, input the validation set into the trained improved YOLOv7s network, and optimize and adjust the model through evaluation indicators; In step 3.6, the test set is input into the optimized improved YOLOv7s network to evaluate the performance of the network.

[0032] Preferably, the forward propagation process during training is as follows: The sample image (monitoring image of each floor of the tool cabinet) first passes through the image initialization layer, is adjusted to a square size using the Padding function, and then is scaled proportionally to 640×640×3 as the input of the backbone network.

[0033] The input of the backbone network passes through the CBS-2 module of the first layer of the backbone and obtains a feature map of 640×640×32 size, the CBS-3 module of the second layer of the backbone and obtains a feature map of 320×320×64 size, the CBS-2 module of the third layer of the backbone and obtains a feature map of 320×320×64 size, the CBS-3 module of the fourth layer of the backbone and obtains a feature map of 160×160×28 size, the ELAN module of the fifth layer of the backbone and obtains a feature map of 160×160×256 size, and the MP-1 module of the sixth layer of the backbone. The module obtains a feature map of 80×80×256 in size, the ELAN module of the backbone seventh layer obtains a feature map of 80×80×512 in size, the MP-1 module of the backbone eighth layer obtains a feature map of 40×40×512 in size, the ELAN module of the backbone ninth layer obtains a feature map of 40×40×1024 in size, the MP-1 module of the backbone tenth layer obtains a feature map of 20×20×1024 in size, and the ELAN module of the backbone eleventh layer obtains a feature map of 20×20×1024 in size, and the initial feature map is extracted from shallow to deep image features.

[0034] The output feature map of the eleventh layer ELAN module of the backbone with a size of 20×20×1024 enters the neck structure, and passes through the SPPCBAM module of the first layer of the neck upsampling branch to obtain a feature map of size 20×20×512, the CBS-1 module of the second layer of the neck upsampling branch to obtain a feature map of size 20×20×256, and the Upsample module of the third layer of the neck upsampling branch to obtain a feature map of size 40×40×256. The Concat module of the fourth layer of the neck upsampling branch is implemented with the back The output feature map of the ELAN module in the ninth layer of the bone is concatenated and fused to obtain a feature map of 40×40×512 size. Then, it is passed through the ELAN-W module in the fifth layer of the neck upsampling branch to obtain a feature map of 40×40×256 size, the CBS-1 module in the sixth layer of the neck upsampling branch has a feature map of 40×40×128 size, and the Upsample module in the seventh layer of the neck upsampling branch has a feature map of 80×80×128 size. The Concat module in the eighth layer of the neck upsampling branch is used to implement the feature map with the ELAN module in the seventh layer of the backbone. The output feature maps of the LAN module are concatenated and fused to obtain a feature map of 80×80×256. Then, they are passed through the ELAN-W module of the ninth layer of the neck upsampling branch to obtain a feature map of 80×80×128 and the MP-2 module of the first layer of the neck downsampling branch to obtain a feature map of 40×40×256. The Concat module of the second layer of the neck downsampling branch realizes the concatenation and fusion operation of the output feature map of the ELAN-W module of the fifth layer of the neck upsampling branch to obtain a feature map of 40×40×512. Then, the Concat module of the second layer of the neck downsampling branch realizes the concatenation and fusion operation of the output feature map of the ELAN-W module of the fifth layer of the neck upsampling branch to obtain a feature map of 40×40×512. The ELAN-W module on the third layer of the neck downsampling branch is used to obtain a feature map of 40×40×256, and the MP-2 module on the fourth layer of the neck downsampling branch is used to obtain a feature map of 20×20×512. The Concat module on the fifth layer of the neck downsampling branch implements the splicing and fusion operation with the output feature map of the SPPCBAM module on the first layer of the neck upsampling branch to obtain a feature map of 20×20×1024. The ELAN-W module on the sixth layer of the neck downsampling branch is then used to obtain a feature map of 20×20×512. Finally, the output feature maps of the ELAN-W module of the ninth layer of the neck upsampling branch, the ELAN-W module of the third layer of the neck downsampling branch, and the ELAN-W module of the sixth layer of the neck downsampling branch are used as inputs of three different scales of the head structure object detection module, and are decoded using three corresponding decoding layers, namely a REP module and a CBM module, to obtain the candidate box position information, object category and confidence.

[0035] Step 4, saving the tool access status detection model and deploying the model, and using the deployed tool status detection model to perform real-time tool access detection, including the following steps: Step 4.1, according to the preset evaluation indicators (intersection-over-union ratio, accuracy, precision, recall rate and F1-score), determine the model with the best evaluation indicators in the verification process; call the torch.save() function to save the parameters of the optimal model into a .pth file, and save the model architecture into a .py file; Step 4.2: In the AI ​​vision-based intelligent safety tool access detection system, use the Python import function to import the saved optimal model, and use the load_state_dict() function to load the saved optimal model parameters to implement the deployment of the optimal model in the intelligent safety tool access detection system. Step 4.3, whenever the cabinet door is closed, the tool access status detection model is linked to realize dynamic detection of the tool access status.

Claims

1. An AI vision-based intelligent safety tool access detection method, characterized in that: include: Step 1, create a data set of tool access status in the tool cabinet; Step 2. Build an improved YOLOv7s network based on the attention mechanism as a tool access status detection model, that is, integrate the hybrid attention mechanism into the SPPCSCP module of the classic YOLOv7s neck structure, and name the module SPPCBAM; Step 3: Divide the data set into training set, validation set and test set according to a certain ratio, and train, validate and perform performance testing on the tool access status detection model; Step 4. Save the tool access status detection model and deploy the model, and use the deployed tool status detection model to perform real-time tool access detection.

2. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 1, it is characterized in that: The Step 1 specifically includes the following steps: Step 1.1, collect monitoring video data of each layer of the tool cabinet: install cameras on each layer of the tool cabinet to achieve full coverage monitoring of each layer of the cabinet and obtain real-time monitoring data; Step 1.2, video frame processing: Use the VideoCapture function in the OpenCV extension library cv2 to perform frame processing on the collected monitoring video data of each layer of the tool cabinet to obtain the original image data set with a resolution of 1280 pixels × 720 pixels; Step 1.3, Dataset annotation: Use the image calibration tool Labelimg to manually calibrate each tool in the image and obtain a standard XML format file. All images and XML format files together constitute the tool access detection dataset.

3. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 1, it is characterized in that: In the Step 2, the network framework of the improved YOLOv7s network based on the attention mechanism is: The improved YOLOv7 model consists of input, image initialization feature extraction layer, backbone network feature extraction module, neck structure feature fusion module and head structure target detection module. The image initialization feature extraction layer is used to preprocess the image and scale the image to a set size.

4. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 3, it is characterized in that: The backbone network feature extraction module is used to extract shallow graphic features and deep semantic features of the image, and transmit three feature maps of different scales to the neck structure. The backbone network is equipped with a CBS-2 module, a CBS-3 module, an ELAN module and an MP-1 module. The ELAN module is a residual structure whose basic modules are CBS-1 and CBS-2, and the MP-1 module is a dual-branch downsampling structure whose basic modules are CBS-1 and CBS-3. The backbone network feature extraction module is used to extract shallow graphic features and deep semantic features of the image, and obtain feature maps of different scales.

5. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 3, it is characterized in that: The neck structure feature fusion module is provided with a CBS-1 module, a SPPCBAM module, an ELAN-W module, and an MP-2 module. The neck structure feature fusion module has an upsampling branch and a downsampling branch, so as to achieve a better fusion of shallow graphic features and deep semantic features, and obtain feature maps of different scales. The ELAN-W module is a residual structure, and the basic modules are CBS-1 and CBS-2; The MP-2 module is a dual-branch downsampling structure, and the basic modules are CBS-1 and CBS-3.

6. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 3, it is characterized in that: The head structure consists of three branches, corresponding to three detection layers of different scales. Each detection layer is responsible for predicting the category and location information of the object. Each branch consists of a REP module and a CBM module.

7. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 5, it is characterized in that: The SPPCBAM module in the neck structure is a spatial pyramid pooling operation that integrates a hybrid attention mechanism, as follows: The input F of the spatial pyramid pooling operation with integrated hybrid attention mechanism is divided into sub-regions of three sizes, generating feature maps F1 with sub-regions of 5×5, F2 with sub-regions of 9×9, and F3 with sub-regions of 13×13 respectively; Perform global average pooling and global maximum pooling on the sub-region of F1 to obtain two feature vectors; pass these two feature vectors through a shared multi-layer perceptron MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F1 channel by channel to generate an enhanced feature map F1'; perform average pooling and maximum pooling on the channel-enhanced feature map F1' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F1' element by element to generate the final enhanced feature map F1"; Perform global average pooling and global maximum pooling on the sub-region of F2 to obtain two feature vectors; pass these two feature vectors through the shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F2 channel by channel to generate an enhanced feature map F2'; perform average pooling and maximum pooling on the channel-enhanced feature map F2' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F2' element by element to generate the final enhanced feature map F2"; Perform global average pooling and global maximum pooling on the sub-region of F3 to obtain two feature vectors; pass these two feature vectors through a shared MLP to generate a channel attention map; multiply the channel attention map with the sub-region of F3 channel by channel to generate an enhanced feature map F3'; perform average pooling and maximum pooling on the channel-enhanced feature map F3' in the channel dimension to obtain two feature maps; connect these two feature maps to form a feature map containing spatial information; apply convolution operation to the connected feature map to generate a spatial attention map; multiply the spatial attention map with the channel-enhanced feature map F3' element by element to generate the final enhanced feature map F3".

8. According to the AI ​​vision-based intelligent safety tool access detection method described in claim 1, it is characterized in that: In the aforementioned Step 3, the tool access status detection model is trained, verified, and tested as follows: Step 3.

1. Set the hyperparameters required for training, including epoch, batch_size, learning rate, training optimizer, and loss function. The loss function is as follows: ； In the formula, , , represents the loss weight, represents the bounding box loss calculated using the IoU loss, represents the confidence loss calculated using binary cross entropy, represents the target detection loss calculated using binary cross entropy; Step 3.2, set model evaluation indicators: select 5 evaluation indicators commonly used in target detection tasks, namely intersection over union ratio, accuracy, precision, recall rate and F1-score; Step 3.3, initialize network parameters; Step 3.4, input the training set into the constructed improved YOLOv7s network, forward propagate the training data in the improved YOLOv7s network, and calculate the loss according to the network output. According to the calculated loss, back propagate and update the network parameters through the gradient descent algorithm until the training is completed; Step 3.5, freeze the network parameters, input the verification set into the trained improved YOLOv7s network, and optimize and adjust the model through evaluation indicators; Step 3.6, input the test set into the optimized improved YOLOv7s network to evaluate the performance of the network.

9. The AI ​​vision-based intelligent safety tool access detection method according to claim 8, characterized in that: The forward propagation process in the training process is as follows: Use the image initialization feature extraction layer to perform preliminary feature extraction on the input image to obtain an initial feature map; Use the backbone network feature extraction module of the improved YOLOv7s network, including the CBS-2 module of the first layer of the backbone network feature extraction module, the CBS-3 module of the second layer of the backbone network feature extraction module, the CBS-2 module of the third layer of the backbone network feature extraction module, the CBS-3 module of the fourth layer of the backbone network feature extraction module, the ELAN module of the fifth layer of the backbone network feature extraction module, the MP-1 module of the sixth layer of the backbone network feature extraction module, the ELAN module of the seventh layer of the backbone network feature extraction module, the MP-1 module of the eighth layer of the backbone network feature extraction module, the ELAN module of the ninth layer of the backbone network feature extraction module, the MP-1 module of the tenth layer of the backbone network feature extraction module, and the ELAN module of the eleventh layer of the backbone network feature extraction module, to extract image features from shallow to deep on the initial feature map; The neck structure feature extraction module using the improved YOLOv7s network includes the SPPCBAM module of the first layer of the sampling branch of the neck structure feature fusion module, the CBS-1 module of the second layer of the sampling branch of the neck structure feature fusion module, the Upsample module of the third layer of the sampling branch of the neck structure feature fusion module, the Concat module of the fourth layer of the sampling branch of the neck structure feature fusion module, the ELAN-W module of the fifth layer of the sampling branch of the neck structure feature fusion module, the CBS-1 module of the sixth layer of the sampling branch of the neck structure feature fusion module, the Upsample module of the seventh layer of the sampling branch of the neck structure feature fusion module, and the The Concat module of the eighth layer of the upsampling branch of the structural feature fusion module, the ELAN-W module of the ninth layer of the upsampling branch of the neck structural feature fusion module, the MP-2 module of the first layer of the downsampling branch of the neck structural feature fusion module, the Concat module of the second layer of the downsampling branch of the neck structural feature fusion module, the ELAN-W module of the third layer of the downsampling branch of the neck structural feature fusion module, the MP-2 module of the fourth layer of the downsampling branch of the neck structural feature fusion module, the Concat module of the fifth layer of the downsampling branch of the neck structural feature fusion module, and the ELAN-W module of the sixth layer of the downsampling branch of the neck structural feature fusion module achieve better fusion of features of different scales; The Concat module of the fourth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN module feature map of the ninth layer of the backbone network feature extraction module; the Concat module of the eighth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN module feature map of the seventh layer of the backbone network feature extraction module; the Concat module of the second layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the ELAN-W module feature map of the fifth layer of the sampling branch of the neck structure feature fusion module; the Concat module of the fifth layer of the sampling branch of the neck structure feature fusion module realizes the splicing and fusion operation with the SPPCBAM module feature map of the first layer of the sampling branch of the neck structure feature fusion module; Finally, the features obtained by the ELAN-W module of the ninth layer of the upsampling branch of the neck structure feature fusion module, the ELAN-W module of the third layer of the downsampling branch of the neck network feature fusion module, and the ELAN-W module of the sixth layer of the downsampling branch of the neck network feature fusion module are used as inputs of three different scales of the head structure target detection module in the improved YOLOv7, and three corresponding decoding layers, namely a REP module and a CBM module, are used for decoding to obtain the candidate box position information, target category and confidence.

10. The AI ​​vision-based intelligent safety tool access detection method according to claim 1, characterized in that: In Step 4, the specific steps include: Step 4.1, save the model framework and model parameters of the tool access status detection model when the evaluation index is optimal during the verification process; Step 4.2, deploy the tool access status detection model in the AI ​​vision-based intelligent safety tool access detection system; Step 4.3, whenever the cabinet door is closed, the tool access status detection model is linked to realize dynamic detection of the tool access status.

Images