Installation and maintenance personnel equipment image detection method and system based on gating kernel self-attention Transform and inverse residual multi-scale convolution module

Abstract

The invention discloses an installation and maintenance personnel equipment image detection method and system based on a gating kernel self-attention Transform and an inverse residual multi-scale convolution module, and relates to the technical field of image detection. The method comprises the following steps: acquiring image data of a worker on an installation and maintenance site, and preprocessing the image data to obtain target image data; performing multi-scale feature information extraction on the target image data by adopting a preset inverse residual multi-scale convolution module to obtain corresponding feature information; on the basis of the feature information, feature coding is carried out by utilizing a gating kernel self-attention Transform, so that corresponding enhanced feature information is obtained; and respectively inputting the enhanced feature information into a plurality of target detection heads with different scales for feature decoding so as to obtain corresponding target categories and corresponding target sizes, and completing image detection. According to the invention, the safety of the current staff can be evaluated efficiently and accurately, the possibility of potential safety hazards is reduced, and the standardization of installation and maintenance staff is improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a method and system for detecting images of equipment of maintenance personnel based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module. Background Art

[0002] It is very important for installation and maintenance personnel to wear safe and standardized equipment for their own safety and customer satisfaction. When installing network equipment, installation and maintenance personnel are in a variety of environments. Installation and maintenance personnel may need to install equipment and deploy cables at high places, or need to install equipment in remote mountainous areas. The actual scenes of installation and maintenance operations vary greatly, and many scenes are outdoors. Installation and maintenance personnel need to wear safe and standardized work equipment. In this context, efficient and accurate installation and maintenance equipment detection plays a preventive role in the personal safety of relevant personnel. At the same time, when facing customers, providing necessary relevant information can effectively improve the standardization and professionalism of installation and maintenance personnel, and gain more trust from customers.

[0003] With the rapid development of artificial intelligence, neural network-based detection models have developed rapidly. Efficient detection networks are very important for mobile equipment. In actual changing scenes, the target changes in the image vary greatly, which is not conducive to recording the information of installation and maintenance personnel. Traditional data recording methods cannot effectively record installation and maintenance equipment. Efficient and convenient compliance detection of installation and maintenance personnel equipment is important.

[0004] Currently, the installation and maintenance quality inspection methods can be divided into installation and maintenance personnel evaluation methods and installation and maintenance inspection methods.

[0005] Prior art 1: "A method and device for evaluating the service portrait of installation and maintenance personnel based on big data analysis". This technology evaluates the service quality of installation and maintenance personnel from six dimensions: basic ability, service timeliness, service standardization, maintenance quality, satisfaction and follow-up support, and establishes a task portrait of installation and maintenance personnel. This method mainly uses big data technology to evaluate the service portrait of installation and maintenance personnel, focusing on the evaluation of installation and maintenance personnel.

[0006] Prior art 2, "AI installation and maintenance inspection method and system based on YOLO algorithm", mainly uses the improved YOLO algorithm to perform compliance inspection on installation and maintenance inspection pictures. The algorithm presets anchor point frames, has a slow training speed, and does not provide specific improvement implementation steps and corresponding example photos.

[0007] Prior art 3, "A method and system for detecting the standardization of installation and maintenance construction based on deep learning", is based on a feature matching algorithm. It performs feature detection on the on-site installation and maintenance construction photos of the inspection project, matches the adaptive feature information, and then scans the feature area of ​​the sample photo based on the detailed feature distribution of the feature to form local feature information. The algorithm uses the extracted local feature information to train the neural network. The algorithm uses a classification detection algorithm to detect the installation and maintenance equipment, and only gives qualified and unqualified detection standards, and cannot provide more detailed detection issues. Summary of the invention

[0008] In order to overcome the above problems or at least partially solve the above problems, the present invention provides a method and system for equipment image detection of installation and maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, which can efficiently and accurately evaluate the safety of the current staff based on the installation and maintenance equipment image data of the current installation and maintenance staff, the inverse residual multi-scale convolution module and the gated kernel attention Transformer, reduce the possibility of safety hazards, and improve the standardization of installation and maintenance personnel.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] In a first aspect, the present invention provides a method for detecting images of equipment of maintenance personnel based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module, comprising the following steps:

[0011] Acquire image data of workers at the installation and maintenance site, and pre-process the image data to obtain target image data;

[0012] The preset inverse residual multi-scale convolution module is used to extract multi-scale feature information of the target image data to obtain corresponding feature information;

[0013] Based on the feature information, the gated kernel self-attention Transformer is used to encode the features to obtain the corresponding enhanced feature information;

[0014] The enhanced feature information is input into multiple target detection heads of different scales for feature decoding to obtain the corresponding target category and corresponding target size to complete image detection.

[0015] Based on the image data of the installation and maintenance equipment of the current installation and maintenance staff, the present invention uses methods such as the inverse residual multi-scale convolution module and the gated kernel attention Transformer to detect whether the current installation and maintenance equipment meets the specifications and meets the safety requirements, improves the standardization and safety awareness of the installation and maintenance staff, and effectively improves the customer's satisfaction with the installation and maintenance process; it can achieve efficient and accurate detection of safety issues of the installation and maintenance staff, reduce the potential safety hazards in the actual work process, improve the standardization of the work, and optimize the actual experience of the customer. This algorithm can cope with complex and changeable installation and maintenance environments without affecting the current installation and maintenance service process.

[0016] Based on the first aspect, further, the method for acquiring image data of workers at the installation and maintenance site and preprocessing the image data to obtain target image data includes the following steps:

[0017] Obtain image data of workers at the installation and maintenance site;

[0018] Determine whether the maintenance work clothes in the image data meet the preset scene clothing requirements. If not, delete the corresponding image data that does not meet the requirements and retain the image data that meets the requirements.

[0019] Mark the target data on the retained image data and record the target position and size;

[0020] The labeled image data is adjusted to a preset image size to obtain the final target image data.

[0021] Based on the first aspect, further, the above-mentioned inverted residual multi-scale convolution module includes point convolution, conventional convolution, hole convolution with different expansion rates and gated convolution; the point convolution and conventional convolution are used for local feature extraction; the hole convolution with different expansion rates is used to extract global feature information; the gated convolution is used to adjust the weights of each feature channel.

[0022] Based on the first aspect, further, the method for extracting multi-scale feature information from target image data using a preset inverse residual multi-scale convolution module comprises the following steps:

[0023] Based on the inverse residual structure of the inverse residual multi-scale convolution module, the feature is dimensionally upgraded and the feature is segmented to obtain a plurality of feature subgraphs; the feature subgraphs include at least four;

[0024] Based on multiple feature subgraphs, multi-scale convolution of the inverse residual multi-scale convolution module is used to extract multi-scale feature information, and feature fusion is performed to obtain fused features;

[0025] Use gated convolution to process the fused features and adjust the weight information of each feature channel to obtain the final aggregated features;

[0026] A multi-scale feature pyramid is constructed based on the inverted residual multi-scale convolution module. Based on the aggregated features, the multi-scale feature pyramid is used for sampling to obtain multiple feature layers.

[0027] Based on multiple feature layers, feature maps are extracted based on the inverse residual multi-scale convolution module to obtain corresponding multiple feature maps.

[0028] Based on the first aspect, further, the method for extracting multi-scale feature information based on multiple feature subgraphs using multi-scale convolution of an inverse residual multi-scale convolution module and fusing features to obtain fused features includes the following steps:

[0029] At least two feature sub-graphs from the multiple feature sub-graphs are input into point convolution and conventional convolution respectively to extract local features and small target features, and the corresponding feature image Y is output. 1 and Y 2 ;

[0030] Input at least two other feature sub-graphs from the multiple feature sub-graphs into two dilation convolutions with different expansion rates to extract large target features and output the corresponding feature image Y 3 and Y 4 ;

[0031] Based on the feature image Y 1 , Y 2 , Y 3 and Y 4 Concatenate features along the channel dimension and obtain the aggregated feature Y through point convolution. C .

[0032] Based on the first aspect, further, the above-mentioned gated core self-attention Transformer includes a gated core self-attention mechanism and a multi-layer perceptron module.

[0033] Based on the first aspect, further, the method for feature encoding using the gated kernel self-attention Transformer includes the following steps:

[0034] Gated kernel self-attention mechanism, which uses the gated kernel module to adjust the weight of the self-attention mask to enhance the important parts of the features;

[0035] Input the attention features into the multi-layer perceptron module to learn the feature parameters;

[0036] Feature encoding is achieved based on the gated kernel self-attention mechanism and multi-layer perceptron module.

[0037] In the second aspect, the present invention provides an image detection system for installation and maintenance personnel equipment based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module, characterized in that it includes a data acquisition and preprocessing module, a feature extraction module, a feature encoding module, and a feature decoding module, wherein:

[0038] The data acquisition and preprocessing module is used to acquire the image data of the workers at the installation and maintenance site and preprocess the image data to obtain the target image data;

[0039] A feature extraction module is used to extract multi-scale feature information from the target image data using a preset inverse residual multi-scale convolution module to obtain corresponding feature information;

[0040] The feature encoding module is used to encode features based on feature information using the gated kernel self-attention Transformer to obtain corresponding enhanced feature information;

[0041] The feature decoding module is used to input the enhanced feature information into multiple target detection heads of different scales for feature decoding, so as to obtain the corresponding target category and the corresponding target size, and complete image detection.

[0042] This system uses the cooperation of multiple modules such as data acquisition and preprocessing module, feature extraction module, feature encoding module and feature decoding module, based on the image data of the installation and maintenance equipment of the current installation and maintenance staff, and uses methods such as inverse residual multi-scale convolution module and gated kernel attention Transformer to detect whether the current installation and maintenance equipment meets the specifications and meets the safety requirements, improve the standardization and safety awareness of the installation and maintenance staff, and effectively improve the customer's satisfaction with the installation and maintenance process; realize efficient and accurate detection of safety issues of installation and maintenance staff, reduce the safety hazards existing in the actual work process, improve the standardization of work, and optimize the actual experience of customers. This algorithm can cope with complex and changeable installation and maintenance environments without affecting the current installation and maintenance service process.

[0043] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0045] The present invention has at least the following advantages or beneficial effects:

[0046] 1. In view of the complex and changeable working environment and great safety hazards of installation and maintenance personnel, an efficient and concise safety detection method is proposed to reduce the safety hazards of actual work and improve the standardization of staff.

[0047] 2. To solve the problem of large changes in target scale in actual scenes, the present invention uses a residual multi-scale convolution module, which can use receptive fields of different scales to extract rich feature information.

[0048] 3. The present invention uses a gated kernel self-attention mechanism to solve the problem of insufficient long-range context extraction capability of the model during model feature encoding.

[0049] 4. To solve the problem of large number of parameters and high complexity of the current model, the model training time of the present invention is short, the number of parameters is smaller, and the prediction accuracy of the model is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 This is a flow chart of a method for detecting images of equipment for maintenance personnel based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module according to an embodiment of the present invention;

[0052] Figure 2 A flowchart of data and preprocessing in an embodiment of the present invention;

[0053] Figure 3 is a flow chart of feature extraction in an embodiment of the present invention;

[0054] Figure 4 A flowchart of feature coding in an embodiment of the present invention;

[0055] Figure 5 A flowchart of feature decoding in an embodiment of the present invention;

[0056] Figure 6 This is an example diagram of collecting staff data in an embodiment of the present invention;

[0057] Figure 7 This is an example diagram of an inverted residual multi-scale convolution module in an embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the KSM-Net model structure in an embodiment of the present invention;

[0059] Fig. 9 Schematic diagram of a gated core self-attention module in an embodiment of the present invention;

[0060] Fig.10 Schematic diagram of model prediction results in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0065] In the description of the embodiments of the present invention, "plurality" means at least 2.

[0066] Example:

[0067] like Figure 1 As shown, in the first aspect, an embodiment of the present invention provides a method for detecting images of equipment for maintenance personnel based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module, comprising the following steps:

[0068] 101. Obtain image data of the staff at the installation and maintenance site, and pre-process the image data to obtain target image data; obtain image data of the staff at the installation and maintenance site, and pre-process the data, delete images that do not match the scene, label the image data, and record the location and size of the target.

[0069] 102. Use the preset inverse residual multi-scale convolution module to extract multi-scale feature information of the target image data to obtain the corresponding feature information; construct an inverse residual multi-scale feature pyramid convolution module to extract multi-scale feature information. The inverse residual multi-scale convolution module uses multiple groups of convolutions with different receptive field sizes to extract feature information of different scales, including point convolution, conventional convolution, dilated convolution with different expansion rates, and gated convolution. Point convolution and conventional convolution are used for local feature extraction; dilated convolution with different expansion rates is used to extract global feature information; gated convolution is used to adjust the weights of each feature channel. The inverse residual multi-scale convolution module constructs a five-layer feature pyramid for feature information encoding.

[0070] 103. Based on the feature information, the gated kernel self-attention transformer is used to encode the feature to obtain the corresponding enhanced feature information; based on the feature information obtained in the above steps, the gated kernel self-attention transformer is used to encode the feature. The gated kernel self-attention transformer includes a gated kernel self-attention mechanism and a multi-layer perceptron module. The gated kernel self-attention mechanism uses the gated kernel module to adjust the weight of the self-attention mask to enhance the important part of the feature. The attention feature is input into the multi-layer perceptron module to learn the feature parameters.

[0071] 104. The enhanced feature information is respectively input into a plurality of different scale object detection heads for feature decoding to obtain the corresponding object category and the corresponding object size, and complete the image detection. The feature information of different scales is obtained by feature encoding of the inverse residual multi-scale convolution module, and after feature enhancement by the gated kernel self-attention module, it is input into three different scale object detection heads. The detection head obtains the category of the object and the corresponding object size.

[0072] Based on the image data of the installation and maintenance equipment of the current installation and maintenance staff, the present invention uses methods such as the inverse residual multi-scale convolution module and the gated kernel attention Transformer to detect whether the current installation and maintenance equipment meets the specifications and meets the safety requirements, improves the standardization and safety awareness of the installation and maintenance staff, and effectively improves the customer's satisfaction with the installation and maintenance process; it can achieve efficient and accurate detection of safety issues of the installation and maintenance staff, reduce the potential safety hazards in the actual work process, improve the standardization of the work, and optimize the actual experience of the customer. This algorithm can cope with complex and changeable installation and maintenance environments without affecting the current installation and maintenance service process.

[0073] Based on the first aspect, further, the method for acquiring image data of workers at the installation and maintenance site and preprocessing the image data to obtain target image data includes the following steps:

[0074] Obtain image data of workers at the installation and maintenance site;

[0075] Determine whether the maintenance work clothes in the image data meet the preset scene clothing requirements. If not, delete the corresponding image data that does not meet the requirements and retain the image data that meets the requirements.

[0076] Mark the target data on the retained image data and record the target position and size;

[0077] The labeled image data is adjusted to a preset image size to obtain the final target image data.

[0078] In some embodiments of the present invention, Figure 2 As shown, the above data acquisition and preprocessing steps mainly include:

[0079] Step 10101, collect images of maintenance workers at the maintenance site. Image data, targets mainly include safety helmets, work clothes, work permits, tool boxes, work cards, shoe covers, dust cloths and tool kits. Work permits and work cards are small target data, and work clothes and dust cloths are large target data. Figure 6 As shown, a partial image of the collected image data is shown.

[0080] Step 10102, delete the images that do not meet the scene requirements. Determine whether the installation and maintenance work clothes in the image meet the current requirements, and delete the images that do not meet the latest work clothes requirements.

[0081] Step 10103: label the image data and record the position and size of the target. Use the labelimg program to label the target data and record the center point position of the target and the size of the graphic box.

[0082] Step 10104, data preprocessing, adjust the image to the specified size. The installation and maintenance personnel dataset includes a variety of different indoor and outdoor installation and maintenance environments, including 8 categories of safety helmets, work clothes, work permits, tool boxes, ID cards, shoe covers, dust cloths and tool kits, with a total of 1030 images. The ratio between the training set, validation set and test set is 8:1:1. The image is adjusted to a size of 640 in length and width. The number of targets in each category of the dataset is shown in the following table:

[0083] category name Number of categories all all 3936 helmet helmet 1080 Work clothes frock 1378 Work permit workcard 340 Toolbox toolbox 241 Work card ID card 250 Shoe covers shoecover 274 Dust cloth dust sheet 121 Toolkit toolbag 252

[0084] Based on the first aspect, further, the above-mentioned inverted residual multi-scale convolution module includes point convolution, conventional convolution, hole convolution with different expansion rates and gated convolution; the point convolution and conventional convolution are used for local feature extraction; the hole convolution with different expansion rates is used to extract global feature information; the gated convolution is used to adjust the weights of each feature channel.

[0085] Based on the first aspect, further, the method for extracting multi-scale feature information from target image data using a preset inverse residual multi-scale convolution module comprises the following steps:

[0086] Based on the inverse residual structure of the inverse residual multi-scale convolution module, the feature is dimensionally upgraded and the feature is segmented to obtain a plurality of feature subgraphs; the feature subgraphs include at least four;

[0087] Based on multiple feature subgraphs, multi-scale convolution of the inverse residual multi-scale convolution module is used to extract multi-scale feature information, and feature fusion is performed to obtain fused features;

[0088] Further, it includes: inputting at least two feature sub-graphs of the plurality of feature sub-graphs into point convolution and conventional convolution respectively, extracting local features and small target features, and outputting corresponding feature images Y 1 and Y 2 ; Input at least two other feature sub-graphs from the multiple feature sub-graphs into two dilation convolutions with different expansion rates to extract large target features and output the corresponding feature image Y 3 and Y 4 ; Based on the feature image Y 1 , Y 2 , Y 3 and Y 4 Concatenate features along the channel dimension and obtain the aggregated feature Y through point convolution. C .

[0089] Use gated convolution to process the fused features and adjust the weight information of each feature channel to obtain the final aggregated features;

[0090] A multi-scale feature pyramid is constructed based on the inverted residual multi-scale convolution module. Based on the aggregated features, the multi-scale feature pyramid is used for sampling to obtain multiple feature layers. Aggregate features of different levels are obtained by the inverted residual multi-scale convolution modules at different stages. These aggregated features are constructed into a feature pyramid to achieve feature fusion of different sizes and enhance the feature extraction capability of the model.

[0091] Based on multiple feature layers, feature maps are extracted based on the inverse residual multi-scale convolution module to obtain corresponding multiple feature maps.

[0092] In some embodiments of the present invention, Figure 3 As shown, the feature extraction step specifically includes:

[0093] Step 10201: The inverted residual structure is used to increase the dimension of the feature and perform feature segmentation. Figure 7 As shown, with the characteristics Figure X As model input, point convolution f 1×1 Perform feature mapping on the feature map along the channel dimension, map the channel dimension C to 4C, expand the size of the channel dimension, and obtain the expanded feature X 4C . Feature X 4C Divide it into 4 parts evenly along the channel dimension to get the feature sub Figure X 1 ,X 2 ,X 3 and X 4 , input into the multi-scale feature extraction module for feature extraction.

[0094] X 4C =f 1×1 (X)

[0095] X 1 ,X 2 ,X 3 ,X 4 =Split(X 4C )

[0096] Step 10202, based on the feature sub-item obtained in step 10201 Figure X 1 ,X 2 ,X 3 and X 4 , the multi-scale convolution module extracts multi-scale feature information. 1 and X 2 are input to the point convolution f 1×1 And the ordinary convolution f 3×3 , used to extract local feature information and small target features, and obtain the output feature Y 1 and Y 2 ;X 3 and X 4 are input to the convolution kernel f' with dilation rates of 2 and 3 respectively. 5×5 and f' 7×7 Extract large target features and get output feature Y 3 and Y 4 , the receptive field of the convolution kernel is equivalent to 5×5 and 7×7. The receptive field sizes of convolution kernels with different dilation rates are different, and dilated convolution does not bring additional parameters. The feature sub-graph Y extracted by the convolution kernel features with different receptive field sizes 1 ,Y 2 ,Y 3and Y 4 , and then concatenate features along the channel dimension, and obtain the aggregated feature Y through point convolution C The above steps are shown in the following formula:

[0097] Y 1 =Silu(BN(f 1×1 (X 1 )))

[0098] Y 2 =Silu(BN(f 3×3 (X 2 )))

[0099] Y 3 =Silu(BN(f' 5×5 (X 3 )))

[0100] Y 4 =Silu(BN(f' 7×7 (X 4 )))

[0101] Y C =f 1×1 (Concat(Y 1 ,Y 2 ,Y 3 ,Y 4 ))

[0102] Step 10203, based on the fusion feature Y obtained in step 10202 C , the gated convolution ψ is used to adjust the features after being aggregated by the multi-scale convolution module. Aggregate feature Y C The weight information of each channel is obtained through global average pooling Gap. After the weight information is adjusted by two linear layers in the channel dimension, the weight is redistributed to each channel. Linear layer 1fc 1 Expand the feature dimension, linear layer 2fc 2 The feature dimensions are restored to obtain the adjusted channel weights. The above steps are shown in the following formula:

[0103] Y=ψ(Y C )

[0104] =Relu(Y C ·Sigmoid(f 1×1 (Gap(Y C ))))

[0105] Step 10204: construct a multi-scale feature pyramid based on the inverse residual multi-scale convolution module. The feature map is first downsampled by Conv1 and Conv2 to obtain the feature layer P1 and P 2 , and then pass through the inverse residual multi-scale convolution modules Ms3, Ms4 and Ms5 to obtain the feature map P 3 ,P 4 and P 5 , the feature map is used for subsequent feature encoding and feature fusion. The above steps are shown in the following formula:

[0106] P 1 =Conv1(X)

[0107] P 2 =Conv2(P 1 )

[0108] P 3 =Ms3(P 2 )

[0109] P 4 = Ms4(P 3 )

[0110] P 5 =Ms5(P 4 )

[0111] Based on the first aspect, further, the method for feature encoding using the gated kernel self-attention Transformer includes the following steps:

[0112] Gated kernel self-attention mechanism, which uses the gated kernel module to adjust the weight of the self-attention mask to enhance the important parts of the features;

[0113] Input the attention features into the multi-layer perceptron module to learn the feature parameters;

[0114] Feature encoding is achieved based on the gated kernel self-attention mechanism and multi-layer perceptron module.

[0115] In some embodiments of the present invention, feature extraction is implemented based on a feature extraction module, which mainly includes a gated core self-attention transformer and a feature fusion module. The gated core self-attention transformer includes a gated core self-attention mechanism and a multi-layer perceptron module. The gated core self-attention mechanism uses a gated core module to adjust the weight of the self-attention mask to enhance the important part of the feature. The attention feature is input into the multi-layer perceptron module to learn the feature parameters. The feature fusion module aggregates features of different scales. Figure 4 As shown, the specific steps of feature encoding include:

[0116] Step 10301: gated convolution adjusts the attribute dimension weights of sequence features. Fig. 9 As shown, based on the feature layer P obtained by the feature extraction module 1025 Long-range semantic feature encoding is performed, and the gated kernel module adjusts the channel dimension weight of the feature layer P5. 5 After linear layer dimension adjustment, it is split along the channel dimension to obtain X 5 and X 6 .X 5 The gated kernel module KGAtten includes gated convolution ψ and kernel function The input features are passed through the linear layer fc to obtain the query vector Q, key vector K and value vector V. The gated convolution first uses the global average pooling Gap to obtain the global aggregation information of the attribute features of the three vectors. The linear function learns the weights of the aggregation information and adjusts the weights of each feature from the attribute dimension. The feature vector obtained after the gated convolution is Q G ,K G and V G The above steps are shown in the following formula:

[0117] X 5 ,X 6 =Split(f 1×1 (P 5 ))

[0118] Q,K,V=Split(f 1×1 (X 5 ))

[0119] Q G =ψ(Q)=Relu(Q·Sigmoid(f 1×1 (Gap(Q))))

[0120] K G =ψ(K)=Relu(K·Sigmoid(f 1×1 (Gap(K))))

[0121] V G =ψ(V)=Relu(V·Sigmoid(f 1×1 (Gap(V))))

[0122] Step 10302, kernel function The feature information of each pixel is enhanced. The kernel function calculates the p-th power feature of each element. The feature with a larger feature information weight originally has a larger feature weight after feature enhancement, which enhances the similarity calculation of features within the class. p Represents the pth power of each element in the matrix. ||Q G || F and represents the Frobenius norm before and after feature enhancement, ||QG || F and The ratio of represents the feature weight. The above steps are shown in the following formula:

[0123]

[0124] Step 10303, based on the query vector Q obtained in step 10302 G , key vector K G Sum value vector V G , calculate the self-attention mask. The self-attention mechanism calculates the correlation between each pixel and readjusts the weight of the feature. The query vector Q G and the key vector K G The self-attention mask X is obtained by matrix outer product Mask , obtain the correlation weights between each sequence. Value vector V G and the self-attention mask X Atten Perform the outer product and reassign the correlation information to each sequence. The above steps are shown in the following formula:

[0125] X Mask =Softmax(Q T K K K )

[0126] X Atten =f 3×3 (X Mask )V G +f 3×3 (X)

[0127] Step 10304, based on the self-attention mask X obtained in step 10303 Atten , the multi-layer perceptron Mlp performs further feature learning, adjusts the information of each feature pixel point, and enhances the nonlinear ability of the model. The above steps are shown in the following formula:

[0128] X KGAtten =Mlp(X Atten )

[0129] Step 10305: residual connection and feature fusion. Based on the self-attention mask X obtained by the feature extraction module 10304 KGAtten With the initial feature X 5 Perform residual connection addition, X 6 The features obtained along the channel dimension and through the gated kernel self-attention module are fused and input into the next layer of features for feature fusion. The above steps are shown in the following formula:

[0130] C 5 =f 1×1(Concat(Mlp((KGAtten(X 5 )+X 5 )+X 5 ),X 6 ))

[0131] In some embodiments of the present invention, Figure 5 As shown, the steps of feature decoding include:

[0132] Step 10401: The feature fusion module fuses features of different scales. 3 ,P 4 And C is obtained based on the feature encoding module 103 5 Perform feature encoding, such as Figure 8 As shown in Figure 2, the feature fusion layer fuses feature maps of different scales. Feature map C 5 After the nearest neighbor difference is upsampled UpSample to the feature map P 4 Same size, along the channel dimension and feature map P 4 After concatenation, the input is sent to the convolutional layer for feature fusion. Feature layer C 4 and feature layer P 3 Use the same method to fuse and get the fused feature C 3 and C 4 The above steps are shown in the following formula:

[0133] C 4 =Conv(Concat(UpSample(C 5 ),P 4 ))

[0134] C 3 =Conv(Concat(UpSample(C 4 ),P 3 ))

[0135] Feature Layer C 3 Renamed to D 3 , feature layer D 3 Then downsample and feature map C 4 Perform feature fusion to obtain D 4 Features, feature map D 4 Then with C 5 Features are fused. Feature layer D 3 ,D 4 and D 5 The input is sent to the detection head to predict the location and category of the target. The above steps are shown in the following formula:

[0136] D 3 =C3

[0137] D 4 =Conv(Concat(Conv(D 3 ),C 4 ))

[0138] D 5 =Conv(Concat(Conv(D 4 ),C 5 ))

[0139] Step 10402, training model. Based on the detection model proposed by the feature extraction module 102 and the feature encoding module 103, this step uses the installation and maintenance personnel data to verify the detection model effect of the gated kernel self-attention Transformer and the inverted residual multi-scale convolution module. The specific method is as follows:

[0140] 1) Obtain images of maintenance personnel in various indoor and outdoor scenes. The images mainly include the personnel's work permit, work clothes, and maintenance equipment. Each image is processed into 640x640 pixels.

[0141] 2) Use the inverted residual multi-scale module to construct a five-layer feature pyramid module to extract feature information of different scales.

[0142] 3) The gated kernel self-attention Transformer is used to obtain high-level global semantic information and enhance the pixel weights in the self-attention mask.

[0143] 4) The fused features are input into the detection head to detect the category and size in the image.

[0144] 5) Determine the training parameters of the model. The ratio between the training set, validation set, and test set is 8:1:1. During training, the model uses the AdamW optimizer to iteratively update the parameters of the model, with a learning rate of 0.001, 200 iterations, and a batch size of 96.

[0145] 6) Model pre-training: The model is pre-trained using the PASCAL VOC2007 and 2012 datasets, and the pre-training parameters are saved.

[0146] 7) Fine-tuning model parameters: During the training process, the model loads the pre-trained parameters and performs model fine-tuning.

[0147] 8) Repeat steps 5) to 7) until the model's test set achieves the best result.

[0148] Step 10403, verify the model. Based on the optimal model parameters obtained in step 10402 of training the model, verify the effect of the model on the test set. The gated kernel self-attention Transformer and the inverted residual multi-scale convolution module of this embodiment are denoted as KSM-Net. In order to prove the effectiveness of the method proposed in this embodiment under the dimensional image data, this example designs an inverted residual structure that only replaces the multi-scale convolution module, and denoted it as DR-Net. Fig.10 The actual prediction results of the model are shown in Figure 2. The recognition effect of all categories under the DR-Net and KSM-Net models, and the model test results are shown below:

[0149]

[0150] The overall recognition rate of the KSM-Net model is higher, with Ap50 and Ap50-95 increased by 3.3% and 2.5% respectively, and the inverted residual multi-scale convolution module has a stronger feature extraction capability. Comparing the two categories of work permits and work cards, under the Ap50 indicator, the work permits and work cards increased by 1.7% and 0.6% respectively; under the Ap50-95 indicator, the two categories increased by 1.7% and 3.1% respectively. The residual multi-scale convolution module in the KSM-Net model can effectively improve the recognition effect of small target categories. At the same time, compared with the large target categories of work clothes and dustproof cloth, under the Ap50 indicator, the effect of the KSM-Net model increased by 1.6% and 8.4%. In summary, the inverted residual multi-scale convolution module used in this implementation example can effectively improve the features of targets of different scales. In order to compare the effectiveness of the gated kernel self-attention mechanism, this embodiment designs a self-attention ablation experiment, and the experimental results are as follows:

[0151] Model Parameter quantity (M) FLOPs(G) Ap50 Ap50-95 KSM-Net 2.96 9.5 73.9 56.9 +SA +0.79 +0.4 75.7 57.7 +KGA +0.25 +0.2 77.2 59.4

[0152] The gated kernel attention mechanism is denoted as KGA, and the standard multi-head self-attention mechanism is denoted as SA. The basic model is KSM-Net, with 2.96M parameters, 9.5G model complexity, 73.9% accuracy of Ap50, and 56.9% accuracy of Ap50-95. After adding SA to the high-level features of the model, the model's Ap50 index increased by 1.8%, and the Ap50-95 index increased by 0.8%; after adding the KGA gated kernel self-attention mechanism, the model's accuracy increased by 3.3%, and Ap50-95 increased by 2.5%. Compared with SA, KGA's parameter volume is reduced by 0.54M, Ap50 increased by 1.5%, and Ap50-95 increased by 1.7%. The self-attention mechanism can effectively improve model performance. Compared with the standard attention mechanism, the KGA module has a smaller model complexity and a stronger recognition effect.

[0153] In the second aspect, an embodiment of the present invention provides an image detection system for maintenance personnel equipment based on a gated kernel self-attention Transformer and an inverse residual multi-scale convolution module, characterized in that it includes a data acquisition and preprocessing module, a feature extraction module, a feature encoding module, and a feature decoding module, wherein:

[0154] The data acquisition and preprocessing module is used to acquire the image data of the workers at the installation and maintenance site and preprocess the image data to obtain the target image data;

[0155] A feature extraction module is used to extract multi-scale feature information from the target image data using a preset inverse residual multi-scale convolution module to obtain corresponding feature information;

[0156] The feature encoding module is used to encode features based on feature information using the gated kernel self-attention Transformer to obtain corresponding enhanced feature information;

[0157] The feature decoding module is used to input the enhanced feature information into multiple target detection heads of different scales for feature decoding, so as to obtain the corresponding target category and the corresponding target size, and complete image detection.

[0158] This system uses the cooperation of multiple modules such as data acquisition and preprocessing module, feature extraction module, feature encoding module and feature decoding module, based on the image data of the installation and maintenance equipment of the current installation and maintenance staff, and uses methods such as inverse residual multi-scale convolution module and gated kernel attention Transformer to detect whether the current installation and maintenance equipment meets the specifications and meets the safety requirements, improve the standardization and safety awareness of the installation and maintenance staff, and effectively improve the customer's satisfaction with the installation and maintenance process; realize efficient and accurate detection of safety issues of installation and maintenance staff, reduce the safety hazards existing in the actual work process, improve the standardization of work, and optimize the actual experience of customers. This algorithm can cope with complex and changeable installation and maintenance environments without affecting the current installation and maintenance service process.

[0159] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing one or more programs and a processor. When the one or more programs are executed by the processor, any method in the first aspect is implemented.

[0160] It also includes a communication interface, and the memory, processor and communication interface are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by executing the software programs and modules stored in the memory. The communication interface can be used to communicate signaling or data with other node devices.

[0161] Among them, the memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0162] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0163] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0165] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0166] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0167] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for detecting equipment images of maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The following steps are involved: Acquire image data of workers at the installation and maintenance site, and pre-process the image data to obtain target image data; The preset inverse residual multi-scale convolution module is used to extract multi-scale feature information of the target image data to obtain corresponding feature information; Based on the feature information, the gated kernel self-attention Transformer is used to encode the features to obtain the corresponding enhanced feature information; The enhanced feature information is input into multiple target detection heads of different scales for feature decoding to obtain the corresponding target category and corresponding target size to complete image detection.

2. According to claim 1, a method for detecting images of equipment for maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The method of acquiring image data of workers at the installation and maintenance site and preprocessing the image data to obtain target image data comprises the following steps: Obtain image data of workers at the installation and maintenance site; Determine whether the maintenance work clothes in the image data meet the preset scene clothing requirements. If not, delete the corresponding image data that does not meet the requirements and retain the image data that meets the requirements. Mark the target data on the retained image data and record the target position and size; The labeled image data is adjusted to a preset image size to obtain the final target image data.

3. According to claim 1, a method for detecting images of equipment for maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The inverse residual multi-scale convolution module includes point convolution, conventional convolution, dilated convolution with different expansion rates and gated convolution; the point convolution and conventional convolution are used for local feature extraction; the dilated convolution with different expansion rates is used to extract global feature information; the gated convolution is used to adjust the weights of each feature channel.

4. According to claim 3, a method for detecting images of maintenance personnel and equipment based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The method for extracting multi-scale feature information from target image data using a preset inverse residual multi-scale convolution module comprises the following steps: Based on the inverse residual structure of the inverse residual multi-scale convolution module, the feature is dimensionally upgraded and the feature is segmented to obtain a plurality of feature subgraphs; the feature subgraphs include at least four; Based on multiple feature subgraphs, multi-scale convolution of the inverse residual multi-scale convolution module is used to extract multi-scale feature information, and feature fusion is performed to obtain fused features; Use gated convolution to process the fused features and adjust the weight information of each feature channel to obtain the final aggregated features; A multi-scale feature pyramid is constructed based on the inverted residual multi-scale convolution module. Based on the aggregated features, the multi-scale feature pyramid is used for sampling to obtain multiple feature layers. Based on multiple feature layers, feature maps are extracted based on the inverse residual multi-scale convolution module to obtain corresponding multiple feature maps.

5. According to claim 4, a method for detecting images of equipment for maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The method for extracting multi-scale feature information based on multiple feature subgraphs using multi-scale convolution of an inverse residual multi-scale convolution module and fusing features to obtain fused features includes the following steps: Input at least two feature sub-graphs from the multiple feature sub-graphs into point convolution and conventional convolution respectively, extract local features and small target features, and output corresponding feature images Y1 and Y2; Input at least two other feature sub-graphs of the multiple feature sub-graphs into two dilation convolutions with different expansion rates to extract large target features, and output corresponding feature images Y3 and Y4; Based on the feature images Y1, Y2, Y3 and Y4, feature concatenation is performed along the channel dimension, and the aggregated feature Y is obtained through point convolution. C .

6. According to claim 1, a method for detecting images of maintenance personnel and equipment based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The gated core self-attention Transformer includes a gated core self-attention mechanism and a multi-layer perceptron module.

7. According to claim 6, a method for detecting images of equipment for maintenance personnel based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: The method for feature encoding using a gated kernel self-attention Transformer comprises the following steps: Gated kernel self-attention mechanism, which uses the gated kernel module to adjust the weight of the self-attention mask to enhance the important parts of the features; Input the attention features into the multi-layer perceptron module to learn the feature parameters; Feature encoding is achieved based on the gated kernel self-attention mechanism and multi-layer perceptron module.

8. A maintenance personnel equipment image detection system based on gated kernel self-attention Transformer and inverse residual multi-scale convolution module, characterized in that: It includes data acquisition and preprocessing module, feature extraction module, feature encoding module and feature decoding module, among which: The data acquisition and preprocessing module is used to acquire the image data of the workers at the installation and maintenance site and preprocess the image data to obtain the target image data; A feature extraction module is used to extract multi-scale feature information from the target image data using a preset inverse residual multi-scale convolution module to obtain corresponding feature information; The feature encoding module is used to encode features based on feature information using the gated kernel self-attention Transformer to obtain corresponding enhanced feature information; The feature decoding module is used to input the enhanced feature information into multiple target detection heads of different scales for feature decoding, so as to obtain the corresponding target category and the corresponding target size, and complete image detection.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images