Experimental instrument shielding identification method based on improved model YOLOv5G

Abstract

The invention discloses an experimental instrument shielding identification method based on an improved model YOLOv5G, and belongs to the crossing field of computer vision and physical experiment teaching, and the method comprises the following steps: 1, collecting a physical experiment video, and constructing a physical experiment video data set; step 2, constructing an improved model YOLOv5G, wherein the improved model YOLOv5G comprises a double-attention visual feature extraction network, a word bag encoder, a multi-scale feature fusion module SPPCSPC and a cross-modal gated attention network; step 3, optimizing model parameters by minimizing a multi-task loss function, and training an improved model YOLOv5G; and step 4, inputting the instrument image containing the occlusion area into the trained improved model YOLOv5G, and outputting the prediction category, the category confidence and the bounding box of the current instrument. According to the method, detection of the occlusion target in the physical experiment instrument image is realized through the constructed improved model YOLOv5G.

Description

Technical Field

[0001] The present invention belongs to the cross - field of computer vision and physics experiment teaching, and specifically relates to a method for identifying occlusions of experimental instruments based on the improved model YOLOv5G. Background Art

[0002] In the scenarios of physics experiment teaching and scientific research, video automatic analysis technology is widely used for experimental process monitoring, action standardization evaluation, and data collection. However, during the actual shooting process, the movement trajectories of experimental equipment, the hand movements of operators, or other objects often cause dynamic occlusions to key areas. For example, the moment of inertia experimental instrument is occluded by the hand, and the electromagnetic induction coil is partially covered by tools. Existing vision - based algorithms have the following significant limitations in occlusion scenarios:

[0003] First, the accuracy of occlusion scene recognition is insufficient; mainstream single - stage detection models (such as YOLOv5) are prone to losing local features in occlusion scenarios, resulting in missed or misdetected targets. For example, due to directly deleting candidate boxes overlapping with high - score boxes, the misdeletion rate of the traditional NMS algorithm is as high as 32% in the dense instrument occlusion scenario, causing missed detection of key experimental instruments.

[0004] Second, the ability of multi - scale feature fusion is weak; the sizes of physics experimental instruments vary greatly, and the existing single - scale pooling of the SPP module is difficult to effectively extract the global features of small and large targets, resulting in large deviations in occlusion boundary localization. For example, the diameter of the hanging ring of the liquid surface tension instrument is about 5 mm, and the diameter of the spectrometer carrier table is about 20 cm. When extracting the global features of the vernier caliper scale line and the turntable of the moment of inertia instrument, the deviation exceeds 15%.

[0005] Third, there is a contradiction between lightweight and real - time performance; the existing models have a high computational complexity and cannot achieve real - time detection on edge devices, making it difficult to meet the real - time analysis requirements of experimental video streams. For example, the number of parameters of the Salience - DETR model reaches 120 million, and it cannot achieve real - time detection on Jetson Nano, with a frame rate lower than 10 FPS.

[0006] Fourth, the robustness of single - modality dependence is insufficient; existing technologies mostly rely on pure visual input and do not combine the text descriptions in experimental records, and cannot improve the understanding ability of occlusion areas through cross - modality information complementarity.

[0007] The above problems make it difficult for existing systems to meet the high - precision analysis requirements in real - world physics experiment scenarios. For example, the intelligent experiment scoring system may wrongly deduct points due to occlusion misjudgment, and the automated data collection device may record incomplete experimental curves due to occlusion interference. Therefore, there is an urgent need for a technical solution that can effectively solve occlusion interference, fuse multi - modality information, and ensure the robustness of the model. Summary of the Invention

[0008] In order to solve the above problems, the present invention proposes an experimental instrument occlusion recognition method based on the improved model YOLOv5G, which realizes the detection of occluded targets in physical experimental instrument images through the constructed improved model YOLOv5G.

[0009] The technical solutions of the present invention are as follows:

[0010] An experimental instrument occlusion recognition method based on an improved model YOLOv5G includes the following steps:

[0011] Step 1: Collect physical experiment videos and build a physical experiment video dataset;

[0012] Step 2: Build an improved model YOLOv5G, which includes a dual-attention visual feature extraction network, a bag-of-words encoder, a multi-scale feature fusion module SPPCSPC, and a cross-modal gated attention network.

[0013] Step 3: Optimize model parameters by minimizing the multi-task loss function and train the improved model YOLOv5G;

[0014] Step 4: Input the instrument image containing the occluded area into the trained improved model YOLOv5G, and output the predicted category, category confidence, and bounding box of the current instrument.

[0015] Furthermore, in step 1, four physics experiments, namely, the adjustment and use of a protractor, the measurement of rigid body moment of inertia, the measurement of object density, and the measurement and research of liquid surface tension, were filmed in the physics experiment center, and data segmentation, data annotation, and data processing were performed on the videos. The physics experiment video dataset is a paired dataset containing video frame sequences and experimental record texts, and is defined as Among them, V i is the i-th video frame sequence, T i is the i-th experimental record text, the number of video frame sequences corresponds to the number of experimental record texts; N is the total number.

[0016] Furthermore, in step 2, the construction process of each part is as follows:

[0017] Step 2.1: Build a dual-attention visual feature extraction network based on the improved YOLOv5s architecture. The dual-attention visual feature extraction network embeds the CBAM attention mechanism, which includes a channel attention module and a spatial attention module. The channel attention module generates channel weights through global pooling and multi-layer perceptron. The spatial attention module fuses the dual-pooling features and generates spatial weights through 7×7 convolution. The specific formula is:

[0018] F v′ = σ(MLP(AvgPool(F v )) + MLP(MaxPool(F v ))) ⊙ F v ;

[0019] F″ v = σ(Conv 7×7 (Concat(AvgPool(F′ v )); MaxPool(F′ v )))) ⊙ F v ′;

[0020] where F v ′ is the feature map after channel attention weighting; σ(·) is the Sigmoid activation function; MLP(·) is the multi-layer perceptron; AvgPool(·) is the global average pooling; F v is the input video feature map; MaxPool(·) is the global max pooling; ⊙ is the element-wise multiplication; F″ v is the feature map after spatial attention weighting; Conv 7×7 (·) is a 7×7 convolutional layer; Concat(·) is the channel concatenation;

[0021] Step 2.2: Extract text features from the experimental record text using a bag-of-words encoder;

[0022] Define the occlusion-related vocabulary {"occlusion"}, perform binary bag-of-words encoding. If the occlusion vocabulary appears in the text, it is marked as 1, indicating the existence of occlusion, otherwise it is 0, indicating no occlusion, to form a one-dimensional binary text feature vector; associate the one-dimensional binary text feature vector with the image occlusion label. In the image occlusion label, 0 indicates no occlusion and 1 indicates occlusion; if there is a 1 in the one-dimensional binary text feature vector, it is determined that the text description has occlusion and is marked as 1 overall;

[0023] Step 2.3: Build a multi-scale feature fusion module SPPCSPC based on cascaded channel-spatial attention. This module is improved based on the ideas of spatial pyramid pooling and cross-stage connection;

[0024] Step 2.4: Design a cross-modal gated attention network to achieve interactive alignment of image features and text features.

[0025] Furthermore, in the above Step 2.3, the specific working process of the multi-scale feature fusion module is as follows:

[0026] Step 2.3.1: Perform multi-level pooling fusion using spatial pyramid pooling:

[0027] F spp = Concat(MaxPool3×3 (F″ v ), MaxPool 5×5 (F″ v ), MaxPool 9×9 (F″ v ), MaxPool 13×13 (F″ v ));

[0028] Among them, F spp is the feature after spatial pyramid pooling; MaxPool 3×3 (·), MaxPool 5×5 (·), MaxPool 9 ×9 (·), MaxPool 13×13 (·) are respectively used for global max pooling with 3×3, 5×5, 9×9, and 13×13 kernels;

[0029] Step 2.3.2, reducing the computational amount through cross-stage connection:

[0030] F sppcsp = Conv(Concat(F spp , Split(F″ v )));

[0031] Among them, F sppcsp is the multi-scale image feature after being processed by the multi-scale feature fusion module; Conv(·) is the convolutional layer; Split(·) is the channel splitting operation.

[0032] Furthermore, in the said step 2.4, the cross-modal gated attention network includes a two-way cross-attention interaction mechanism, a gated fusion unit, and a multi-modal fusion module. The specific working process is as follows:

[0033] Step 2.4.1, calculating the cross-attention between the image feature and the text feature through the two-way cross-attention interaction mechanism. The formula is:

[0034] Q v = F sppcsp W q ;

[0035]

[0036] Among them, W q , W k , W k are all learnable weight matrices, which are respectively used to generate query, key, and value vectors; T is the transpose symbol; is the scaling factor; Softmax(·) is normalized along the text dimension to generate the attention weight matrix; is a text feature; Q v is the query vector of the image feature; K t is the key vector of the text feature; V t is the value vector of the text feature; Attn v2t is the text feature after final attention weighting;

[0037] Step 2.4.2, dynamically adjust the fusion ratio of the image feature and the text feature through the gated fusion unit, and dynamically adjust the modal contribution weight:

[0038] G = σ(W g ·[Attn v2t ; F sppcsp );

[0039] where, W g is the linear transformation weight matrix; G is the gated weight matrix;

[0040] Step 2.4.3, perform the fusion of the two modal features through the multi-modal fusion module:

[0041] F fusion = G⊙Attn v2t +(1 - G)⊙F sppcsp ;

[0042] where, F fusion is the finally fused feature.

[0043] Furthermore, in the said Step 3, the formula of the multi-task loss function is:

[0044]

[0045] where, is the multi-task loss function, is the objective loss, is the bounding box loss, is the class loss; λ1, λ2, λ3 are different parameters.

[0046] Furthermore, in the said Step 4, the bounding box includes the actual bounding box center coordinates (b x , b y ) and the width and height, and the calculation formula is:

[0047]

[0048] where, b x , b y are respectively the abscissa and ordinate of the actual bounding box center coordinates; b w , b h are respectively the width and height of the actual bounding box; tx , t y are respectively the horizontal offset and the vertical offset of the center point of the current grid; c x , c y are respectively the abscissa and the ordinate of the upper left corner coordinate of the current grid; p w , p h are respectively the width and the height of the preset anchor box; t w , t h are respectively the width and height scaling factors relative to the anchor box size;

[0049] The calculation formula of the class confidence is:

[0050]

[0051] where is the confidence of the th class ; is the object confidence; is the probability of the th class ; is the network prediction value of the th class ; t is the network prediction value;

[0052] Select the class index corresponding to the maximum value from the confidences of all classes as the final predicted class.

[0053] The beneficial technical effects brought by the present invention:

[0054] 1. The ability to recognize occlusion scenes in physical experiment videos is significantly enhanced. The architecture is optimized through the channel spatial attention mechanism (CBAM) and the multi-scale feature fusion with context (SPPCSPC) module. Tests on the made physical experiment dataset PEVD-Occ show that the mAP of occluded object detection reaches 38.7%, which is 15.3 percentage points higher than the original YOLOv5, and the false detection rate is significantly reduced.

[0055] 2. The performance of cross-scale feature fusion is broken through. The SPPCSPC module with gradient pooling kernels changed to 3×3, 5×5, 9×9, 13×13 is added to the backbone network to achieve multi-level receptive field fusion, supporting cross-scale detection from 5mm-level small objects (the hanging ring of the liquid surface tension meter) to 20cm-level large objects (the turntable of the rigid body moment of inertia meter). A pyramid feature space is constructed by using a 4-level parallel pooling structure with gradient pooling kernels of 3×3, 5×5, 9×9, 13×13, which improves the detection accuracy of small objects and reduces the occlusion boundary localization error.

[0056] 3. Co - optimization of lightweight and real - time performance. Through depth - wise separable convolution and channel pruning techniques, the number of parameters is compressed while maintaining the model performance. The training and inference speeds are accelerated compared to other single - stage models.

[0057] 4. Innovation in multi - modal feature alignment. Construct a cross - modal interaction network to achieve visual - text feature space alignment through contrastive learning; use a bag - of - words encoder to extract semantic information from experimental records; design a dual - path cross - attention interaction mechanism to enhance the correlation between image features and text descriptions; adopt a gated attention network to achieve modal interactive alignment and reduce the false detection rate.

[0058] 5. Improvement in teaching scenario adaptability. The present invention can be subsequently integrated into an intelligent scoring system for physical experiments, automatically identifying instrument occlusion errors in students' operations (such as hand occlusion during the adjustment of the spectrometer slit), and assisting in correcting experimental data.

[0059] 6. The present invention has been successfully applied to four types of physical experiment teaching scenarios, namely the experiment on the adjustment and use of a dividing head, the experiment on measuring the moment of inertia of a rigid body, the experiment on measuring the density of an object, and the experiment on measuring and studying the surface tension of a liquid. The cumulative processing time of experimental videos exceeds 1200 hours. The stability rate of instrument recognition under occlusion interference has been significantly improved, which can effectively support educational informatization application scenarios such as physical experiment scoring. Description of the Drawings

[0060] Figure 1 It is a flowchart of the experimental instrument occlusion recognition method based on the improved model YOLOv5G of the present invention.

[0061] Figure 2 It is an overall architecture diagram of the improved model YOLOv5G of the present invention.

[0062] Figure 3 It is a multi - task loss function curve graph of the training set and validation set of the improved model YOLOv5G in the experiment of the present invention after 400 rounds of iteration.

[0063] Figure 4 It is a graph of the change of the multi - task loss function during the training process of each model in the experiment of the present invention.

[0064] Figure 5 It is a visualization result graph of the occlusion detection of the adjustment and use of the spectrometer in the experiment of the present invention; among them, (a), (b), (c), and (d) respectively correspond to the visualization result graphs of YOLOv5G, YOLOv5, YOLOv8, and YOLOv9.

[0065] Figure 6This is the visualization result diagram for the determination and research of liquid surface tension in the experiments of the present invention; among them, (e), (f), (g), and (h) respectively correspond to the visualization result diagrams of YOLOv5G, YOLOv5, YOLOv8, and YOLOv9. Detailed implementation manner

[0066] The present invention will be further described in detail below in conjunction with the accompanying drawings and the specific implementation manner:

[0067] As Figure 1 shown, an occlusion recognition method for experimental instruments based on the improved model YOLOv5G includes the following steps:

[0068] Step 1, collect physical experiment videos and construct a physical experiment video dataset;

[0069] Videos of 4 physical experiments, namely the adjustment and use of a graduated meter, the determination of the moment of inertia of a rigid body, the determination of the density of an object, and the determination and research of liquid surface tension, were filmed in the school's physical experiment center. When filming, issues such as lighting and angle were considered to ensure the diversity of the samples. Through data segmentation, data annotation, and data processing of the 4 physical experiment videos, a dataset of 3902 occluded pictures was finally determined. The annotation information includes instrument type, spatial coordinates, and occlusion status, forming structured metadata; for the occlusion status, there are two states: unoccluded and occluded. The physical experiment video dataset is a paired dataset containing video frame sequences and experimental record texts, defined as where, V i is the i-th video frame sequence, T i is the i-th experimental record text, and the number of video frame sequences corresponds to the number of experimental record texts; N is the total number.

[0070] Step 2, construct the improved model YOLOv5G. The improved model YOLOv5G includes a dual-attention visual feature extraction network, a bag-of-words encoder, a multi-scale feature fusion module SPPCSPC, and a cross-modal gated attention network; the construction process of each part is as follows:

[0071] Step 2.1, construct a dual-attention visual feature extraction network based on the improved YOLOv5s architecture to extract image features. The CBAM attention mechanism is embedded in the dual-attention visual feature extraction network. The CBAM attention mechanism includes a channel attention module and a spatial attention module; the channel attention module generates channel weights through global pooling and a multi-layer perceptron, and the spatial attention module generates spatial weights after fusing dual-pooling features through a 7×7 convolution; the specific formula is:

[0072] F v ′ = σ(MLP(AvgPool(F v)) + MLP(MaxPool(F v ))) ⊙ F v ;

[0073] F″ v = σ(Conv 7×7 (Concat(AvgPool(F′ v )); MaxPool(F′ v )))) ⊙ F v ′;

[0074] where F v ′ is the feature map after channel attention weighting; σ(·) is the Sigmoid activation function that maps the weights to the interval [0, 1]; MLP(·) is a multi-layer perceptron used to learn the non-linear relationship between channels; AvgPool(·) is global average pooling that compresses each channel of the feature map into a scalar; F v is the input video feature map with dimensions [C, H, W], where C, H, and W are the number of channels, height, and width respectively. MaxPool(·) is global max pooling used to extract the maximum response value of each channel. ⊙ is element-wise multiplication used to perform channel-level weighting on the original feature map. F″ v is the feature map after spatial attention weighting; Conv 7×7 (·) is a 7×7 convolutional layer used to fuse spatial information to generate an attention weight map. Concat(·) is channel concatenation used to concatenate the two pooling results.

[0075] Step 2.2: Extract text features from the experimental record text using a bag-of-words encoder;

[0076] Define the occlusion-related vocabulary {"occlusion"}, perform binary bag-of-words encoding. If the occlusion vocabulary appears in the text, it is marked as 1 (indicating the presence of occlusion), otherwise it is 0 (indicating no occlusion), forming a one-dimensional binary text feature vector. Associate the one-dimensional binary text feature vector with the image occlusion label (0 = no occlusion, 1 = occlusion). If there is a 1 in the one-dimensional binary text feature vector, it is determined that the text description has occlusion (the overall label is 1).

[0077] Step 2.3: Construct a multi-scale feature fusion module SPPCSPC based on cascaded channel-spatial attention. This module is an improved module based on the ideas of spatial pyramid pooling (SPP) and cross-stage connection (CSP), used to extract multi-scale image features, reduce the computational amount, achieve multi-level pooling, and finally generate pyramid features by parallel execution of max pooling and subsequent concatenation; the specific working process is as follows:

[0078] Step 2.3.1: Perform multi-level pooling fusion using spatial pyramid pooling:

[0079] F spp = Concat(MaxPool 3×3 (F″ v ), MaxPool 5×5 (F″ v ), MaxPool 9×9 (F″ v ), MaxPool 13×13 (F″ v ));

[0080] Among them, F spp is the feature after spatial pyramid pooling; MaxPool 3×3 (·), MaxPool 5×5 (·), MaxPool 9 ×9 (·), MaxPool 13×13 (·) are respectively performing global max pooling using 3×3, 5×5, 9×9, 13×13 kernels;

[0081] Step 2.3.2, reducing the computational complexity through cross-stage connection:

[0082] F sppcsp = Conv(Concat(F spp , Split(F″ v )));

[0083] Among them, F sppcsp is the multi-scale image feature after being processed by the multi-scale feature fusion module; Conv(·) is the convolutional layer; Split(·) is the channel splitting operation; Split(F″ v ) means splitting the channel dimension of F″ v into two parts through the channel splitting operation.

[0084] Step 2.4, designing a cross-modal gated attention network to achieve interactive alignment of image features and text features, that is, the cross-attention weights need to focus on semantically related regions (such as occluded image regions and occluded text words). The cross-modal gated attention network includes a two-way cross-attention interaction mechanism, a gated fusion unit, and a multi-modal fusion module. The specific working process is as follows:

[0085] Step 2.4.1, calculating the cross-attention of image features and text features through the two-way cross-attention interaction mechanism. The formula is:

[0086] Q v = F sppcsp W q ;

[0087]

[0088] Among them, W q , W k , W k are all learnable weight matrices, which are used to generate query, key, and value vectors respectively. T is the transpose symbol; is the similarity matrix between the visual query and the text key; is the scaling factor; Softmax(·) normalizes along the text dimension to generate the attention weight matrix. is the text feature; Q v is the query vector of the image feature; K t is the key vector of the text feature; V t is the value vector of the text feature; Attn v2t is the finally attention-weighted text feature.

[0089] Step 2.4.2: Dynamically adjust the fusion ratio of the image feature and the text feature through the gated fusion unit to suppress noise and dynamically adjust the modal contribution weight:

[0090] G = σ(W g ·[Attn v2t ; F sppcsp );

[0091] Among them, W g is the linear transformation weight matrix; G is the gated weight matrix;

[0092] Step 2.4.3: Perform the fusion of the two modal features through the multimodal fusion module:

[0093] F fusion = G⊙Attn v2t +(1 - G)⊙F sppcsp ;

[0094] Among them, F fusion is the finally fused feature.

[0095] Step 3: Optimize the model parameters by minimizing the multitask loss function to train the improved model YOLOv5G;

[0096] The training of the improved model YOLOv5G adopts the multitask loss function:

[0097]

[0098] Among them, is the multitask loss function, is the objectness loss, is the bounding box loss, is the class loss; λ1, λ2, and λ3 are different parameters.

[0099] Step 4: Input the instrument image containing the occluded area into the trained improved model YOLOv5G, and output the predicted class, class confidence, and bounding box of the current instrument. The architecture of the improved model YOLOv5G is as Figure 2 shown, including a backbone network, a neck network, and a head network.

[0100] In the backbone network, the input instrument image with the occluded area first enters the CBS (Convolution + BatchNorm + SILU, convolution, batch normalization, and SILU activation function) module for convolution, batch normalization, and SILU activation to extract initial features; then enters another CBS module for convolution, batch normalization, and activation to further extract features; enters the CSP1_1 (where CSP is Cross Stage Partial connections) module for feature extraction; the CSP module is divided into CSP1_x and CSP2_x; CSP1_x is mainly used for feature extraction, while CSP2_x focuses more on feature fusion, and x represents the number of repetitions of the Bottleneck layer in this module; then enters a CBS module to continue convolution, batch normalization, and activation to extract deeper features; enters the CSP1_2 module to further extract and fuse features to obtain text features; enters a CBS module to continue convolution, batch normalization, and activation to extract deeper features; enters the CSP1_3 module to further extract and fuse features to obtain text features; enters a CBS module to continue convolution, batch normalization, and activation to extract deeper features; enters the CSP1_1 module to further extract and fuse features; finally enters the SPPF (Spatial Pyramid Pooling Fast) module to perform consecutive pooling operations to achieve multi-scale feature fusion.

[0101] The neck network obtains text features from the features of the SPPF module through CBAM (Convolutional Block Attention Module). CBAM contains two concatenated attention sub-modules, which act on the channel and spatial dimensions respectively, learn the importance weights of different channels, enhance the feature responses of important channels, learn the importance weights of spatial positions, and enhance the feature responses of key regions; fuse the feature map with the text features and input them into SPPCSPC (Spatial Pyramid Pooling-Cross Stage Partial Connections) to further fuse and enhance the feature map; then enter a CBS module for convolution, batch normalization, and activation to extract deep features and obtain a low-resolution feature map a; upsample the low-resolution feature map a to obtain a high-resolution feature map a for fusion with higher-level feature maps; concatenate the obtained high-resolution feature map a with the text features obtained through the backbone network CSP1_3 module to fuse multi-scale features; then enter the CSP2_1 module to further extract and fuse features; then enter a CBS module to continue convolution, batch normalization, and activation to extract deeper features and obtain a low-resolution feature map b; upsample the low-resolution feature map b to obtain a high-resolution feature map b; concatenate the obtained high-resolution feature map b with the text features obtained through the backbone network CSP1_2 module; enter the CSP2_1 module to further extract and fuse features to obtain features Figure 1 ; Features Figure 1 Enter the detection head and perform convolution, batch normalization, and activation again to extract deeper features; concatenate it with the high-resolution feature map b; enter the CSP2_1 module to further extract and fuse features to obtain features Figure 2 ; Features Figure 2 Enter the detection head and perform convolution, batch normalization, and activation again to extract deeper features; concatenate it with the feature map a; enter the CSP2_1 module to further extract and fuse features to obtain features Figure 3 ; Features Figure 3 Enter the detection head.

[0102] In the head network, three detection heads respectively receive fused feature maps of three different scales. The feature maps are evenly divided into several small grids according to their resolutions, and each grid has three preset anchor boxes. The detection heads use 1×1 convolutional kernels to perform convolutional operations on the entire feature map. The 1×1 convolutional kernels are used to generate prediction parameters for each anchor box in each grid. The single-anchor box prediction parameters include the offset relative to the center point of the current grid, the width and height scaling factors relative to the anchor box size, the unnormalized object confidence, and the probabilities of various categories, which are converted into actual bounding box coordinates, object confidence, and probabilities of various categories through mathematical transformations. The anchor box is a predefined standard rectangular box used to predict the actual bounding box of the target. Each anchor box has a fixed ratio and size, which helps to improve the detection accuracy of the model for targets of different shapes and sizes. The specific sizes of the anchor boxes are obtained by the model through clustering analysis of a large number of labeled data, aiming to cover as many different shapes and sizes of various targets as possible. A total of nine anchor boxes are generated, and the nine anchor boxes are evenly distributed to the feature maps of the large, medium, and small three different scales according to certain rules.

[0103] The actual bounding box is obtained by using bounding box coordinate transformation. The actual bounding box includes the center coordinates (b x ,b y ) and the width and height. The calculation formula is as follows:

[0104]

[0105] where b x , b y are the abscissa and ordinate of the center coordinates of the actual bounding box respectively; b w , b h are the width and height of the actual bounding box respectively; σ(·) is the Sigmoid function, which limits the coordinate offset to [0,1] to ensure that the center falls within the current grid; t x , t y are the horizontal offset and vertical offset of the center point of the current grid respectively; c x , c y are the abscissa and ordinate of the upper left corner coordinates of the current grid respectively; p w , p h are the width and height of the preset anchor box respectively; t w , t h are the width and height scaling factors relative to the anchor box size respectively;

[0106] The calculation formula for the object confidence is as follows:

[0107]

[0108] where t is the network prediction value, that is, the probability that the predicted bounding box contains an object;

[0109] The calculation formula for the probabilities of various categories is as follows:

[0110]

[0111] where is the probability of the th category ; is the network prediction value of the th category ;

[0112] Finally, the confidence of each category is calculated, and the formula is:

[0113]

[0114] where is the confidence of the th category ;

[0115] The conditional probabilities of object existence and belonging to a certain category are considered simultaneously.

[0116] The bounding box coordinates, object confidence, and category probabilities are processed through threshold screening and Soft-NMS (Soft-NMS) to obtain the final detection results. Threshold screening selects the category index corresponding to the maximum value from the confidences of all categories as the final predicted category. If the maximum value is lower than the preset threshold of 0.3, the detection box is discarded, indicating that it does not contain a valid target. Soft-NMS retains reliable predictions and alleviates the problem of missed detections in traditional NMS. Especially for the detection of dense or overlapping targets, for the detection boxes of the same category, after sorting by confidence, the box with the highest score is retained, and the confidences of other boxes with a high overlap (IoU) with it are attenuated, rather than directly eliminating them. Finally, the boxes with attenuated confidences still higher than the threshold of 0.25 are retained.

[0117] To prove the feasibility and superiority of the present invention, experiments of the improved model YOLOv5G of the present invention on the physical experiment dataset produced are given.

[0118] Figure 3 is the multi-task loss function curve graph of the training set and the validation set after the improved model YOLOv5G has undergone 400 rounds of iteration. The bounding box loss (box_loss) measures the matching degree between the predicted box and the true box. The objectness loss (obj_loss) measures whether the model can correctly predict the existence of a target. The class loss (cls_loss) measures the loss of the model's prediction of the target category. If the training loss is very low and the validation loss is very high, it indicates that the model is overfitting. If both are very high, it indicates that the model is underfitting. Observe Figure 3It can be significantly seen that the loss functions of the training set and the validation set of the improved model YOLOv5G of the present invention tend to be flat after 400 rounds of iteration, and it has been trained to the best state.

[0119] Figure 4 It is a graph showing the change of the multi-task loss function of the model during 100 rounds of training for all data. It can be clearly seen from the graph that the loss function of the improved model YOLOv5G of the present invention decreases faster than that of YOLOv5, YOLOv8, and Salience DETR. During the training process, the training speed of YOLOv5G is relatively fast, and the training time is 4.5 hours, while the training time for Salience DETR to converge is 8 hours.

[0120] Table 1 Comparison experiment result table

[0121]

[0122] The five algorithms of YOLOv5G, YOLOv5, YOLOv8, YOLOv9, and Salience DETR are respectively tested on the physical experiment dataset (PEVD) and the occluded physical experiment dataset (PEVD-Occ) to obtain Table 1. P represents precision, which reflects the model's ability to handle false positives. The higher the proportion of real targets, the higher the precision. R represents recall, which reflects the model's ability to handle missed detections. A high recall means that the model can detect most of the actual targets with fewer false negatives. mAP 50 represents the mean average precision calculated under the condition that the intersection over union (IoU) threshold is 0.5. The threshold of 0.5 means that the overlapping area between the predicted box and the ground truth box should account for at least 50% of the union area to be considered a correct detection. mAP refers to the mean average precision calculated under different IoU thresholds. Here, it represents the mean average precision calculated with IoU ranging from 0.5 to 0.95 in steps of 0.05, which reflects the overall detection ability of the model. A high mAP indicates that the model can maintain a high precision under various IoU thresholds, and has stronger generalization ability and robustness. The bolded values in the table are the optimal results of mAP, and the underlined values are the sub-optimal results of mAP.

[0123] From the comparative experiment results in Table 1, it can be clearly seen that the improved model YOLOv5G of the present invention performs excellently in multiple aspects. The accuracy of the improved model YOLOv5G of the present invention on PEVD is 0.878, which is higher than other models. The proportion of correct detections is the highest, and there are fewer false alarms. The mAP of the improved model YOLOv5G of the present invention is 0.566, which is higher than YOLOv5 (0.444), YOLOv8 (0.468) and YOLOv9 (0.540). This indicates that under all IoU thresholds, the overall detection effect of the improved model YOLOv5G of the present invention is better, with a higher balance of detection accuracy and recall rate. On PEVD-Occ, the mAP of YOLOv5G is 0.387, which is significantly better than YOLOv5 (0.234), YOLOv8 (0.312) and YOLOv9 (0.291). The improved model YOLOv5G of the present invention maintains a high level in most cases, with excellent comprehensive performance of accuracy and speed. It has strong practical application value and wide applicability in the obstacle occlusion target detection task in physical experiment videos.

[0124] Figure 5 The visualization results of occlusion detection for the same picture during the experimental process of the adjustment and use of a spectrometer by different models. Among them, (a), (b), (c), and (d) correspond to the visualization results of YOLOv5G, YOLOv5, YOLOv8, and YOLOv9 respectively. Figure 6 The visualization results of occlusion detection for the same picture during the experiment on the measurement and study of the surface tension of liquids by different models. Among them, (e), (f), (g), and (h) correspond to the visualization results of YOLOv5G, YOLOv5, YOLOv8, and YOLOv9 respectively. The numerical value on the detected picture represents the confidence level of the detection box. This numerical value indicates the probability that the model believes that the target is contained within the detection box. The higher the numerical value, the more confident the model is in the detection result.

[0125] From Figure 5 and Figure 6 the visualization results of occlusion detection in the two experiments, it can be seen that in the same picture, the YOLOV5G model detected all the occluded targets, while there were cases of missed detections in other models. For example, Figure 5 the YOLOV5 model missed detecting the coarse focus adjustment screw and the pen in Figure 6 and the YOLOV5 model missed detecting the pen in Figure 5 Observing most of the occluded targets in Figure 6For all occluded targets, it can be clearly seen that the overall confidence of the YOLOV5G model is relatively high compared to YOLOV5, YOLOV8, and YOLOV9. Secondly, the occlusion detection effect of YOLOV9 is stronger than that of YOLOV8, which is stronger than that of YOLOV5.

[0126] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the scope of the essence of the present invention should also fall within the protection scope of the present invention.

Claims

1. An occlusion recognition method for experimental instruments based on the improved model YOLOv5G, characterized in that, The steps include: Step 1: Collect physical experiment videos and build a physical experiment video dataset; Step 2: Build an improved model YOLOv5G, which includes a dual-attention visual feature extraction network, a bag-of-words encoder, a multi-scale feature fusion module SPPCSPC, and a cross-modal gated attention network. Step 3: Optimize model parameters by minimizing the multi-task loss function and train the improved model YOLOv5G; Step 4: Input the instrument image containing the occluded area into the trained improved model YOLOv5G, and output the predicted category, category confidence, and bounding box of the current instrument.

2. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 1, characterized in that, In step 1, videos of four physical experiments, including the adjustment and use of a protractor, the measurement of rigid body moment of inertia, the measurement of object density, and the measurement and research of liquid surface tension, were filmed at a physical experiment center, and data segmentation, data labeling, and data processing were performed on the videos. The physical experiment video dataset is a paired dataset containing video frame sequences and experimental record texts, defined as i = 1, 2, …, N; Among them, V i is the i-th video frame sequence, T i is the i-th experimental record text, and the number of video frame sequences corresponds to the number of experimental record texts; N is the total number.

3. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 1, wherein, In step 2, the construction process of each part is as follows: Step 2.1: Build a dual-attention visual feature extraction network based on the improved YOLOv5s architecture. The dual-attention visual feature extraction network embeds the CBAM attention mechanism, which includes a channel attention module and a spatial attention module. The channel attention module generates channel weights through global pooling and multi-layer perceptron. The spatial attention module fuses the dual-pooling features and generates spatial weights through 7×7 convolution. The specific formula is: F v ′ = σ(MLP(AvgPool(F v )) + MLP(MaxPool(F v ))) ⊙ F v ; F″ v = σ(Conv 7×7 (Concat(AvgPool(F′ v )); MaxPool(F′ v )))) ⊙ F v ′; Among them, F v ′ is the feature map after channel attention weighting; σ(·) is the Sigmoid activation function; MLP(·) is the multi-layer perceptron; AvgPool(·) is the global average pooling; F v is the input video feature map; MaxPool(·) is the global maximum pooling; ⊙ is the element-wise multiplication; F″ v is the feature map after spatial attention weighting; Conv 7×7 (·) is a 7×7 convolutional layer; Concat(·) is the channel concatenation; Step 2.2: Use the bag-of-words encoder to extract text features from the experimental record text; Define an occlusion-related vocabulary {"occlusion"} and perform binary bag-of-words encoding. If an occlusion word appears in the text, it is marked as 1, indicating the presence of occlusion; otherwise, it is marked as 0, indicating no occlusion, forming a one-dimensional binary feature vector of the text. Associate the one-dimensional binary feature vector of the text with the image occlusion label. In the image occlusion label, 0 indicates no occlusion and 1 indicates occlusion. If there is a 1 in the one-dimensional binary feature vector of the text, it is determined that the text description has occlusion and the entire text is marked as 1. Step 2.3: Construct a multi-scale feature fusion module SPPCSPC based on cascaded channel-spatial attention. This module is an improvement based on the idea of spatial pyramid pooling and cross-stage connection. Step 2.4: Design a cross-modal gated attention network to achieve interactive alignment of image features and text features.

4. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 3, characterized in that, In step 2.3, the specific working process of the multi-scale feature fusion module is as follows: Step 2.3.1: Use spatial pyramid pooling for multi-level pooling fusion: F spp = Concat(MaxPool 3×3 (F″ v ), MaxPool 5×5 (F″ v ), MaxPool 9×9 (F″ v ), MaxPool 13×13 (F″ v )); Among them, F spp is the feature after spatial pyramid pooling; MaxPool 3×3 (·), MaxPool 5×5 (·), MaxPool 9×9 (·), MaxPool 13×13 (·) are respectively used for global max pooling with 3×3, 5×5, 9×9, and 13×13 kernels; Step 2.3.2: Reduce computational complexity through cross-stage connections: F sppcsp = Conv(Concat(F spp , Split(F″ v ))); Among them, F sppcsp is the multi-scale image feature processed by the multi-scale feature fusion module; Conv(·) is the convolutional layer; Split(·) is the channel splitting operation.

5. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 4, characterized in that, In step 2.4, the cross-modal gated attention network includes a two-way cross-attention interaction mechanism, a gated fusion unit, and a multimodal fusion module. The specific working process is as follows: Step 2.4.1: Calculate the cross attention of image features and text features through the two-way cross attention interaction mechanism. The formula is: Q v = F sppcsp W q ; Among them, W q , W k , W k are all learnable weight matrices, which are used to generate query, key, and value vectors respectively; T is the transpose symbol; is the scaling factor; Softmax(·) normalizes along the text dimension to generate the attention weight matrix; is the text feature; Q v is the query vector of the image feature; K t is the key vector of the text feature; V t is the value vector of the text feature; Attn v2t is the finally attention-weighted text feature; Step 2.4.2: Dynamically adjust the fusion ratio of image features and text features through the gated fusion unit, and dynamically adjust the modality contribution weight: G = σ(W g ·[Attn v2t ; F sppcsp ) Among them, W g is the linear transformation weight matrix; G is the gating weight matrix; Step 2.4.3: Fusion of two modal features through the multimodal fusion module: F fusion = G ⊙ Attn v2t + (1 - G) ⊙ F sppcsp ; Among them, F fusion is the finally fused feature.

6. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 5, characterized in that In the above step 3, the formula of the multi-task loss function is as follows: Among them, is the multi-task loss function, is the objective loss, is the bounding box loss, is the class loss; λ1, λ2, and λ3 are different parameters.

7. The experimental instrument occlusion recognition method based on the improved model YOLOv5G according to claim 6, characterized in that, In step 4, the bounding box includes the actual center coordinates of the bounding box (b x , b y ) and the width and height. The calculation formula is as follows: Among them, b x and b y are the abscissa and ordinate of the center coordinates of the actual bounding box respectively; b w and b h are the width and height of the actual bounding box respectively; t x and t y are the horizontal offset and vertical offset of the center point of the current grid respectively; c x and c y are the abscissa and ordinate of the coordinates of the upper left corner of the current grid respectively; p w and p h are the width and height of the preset anchor box respectively; t w and t h are the width and height scaling factors relative to the size of the anchor box respectively; The calculation formula of the class confidence is: Among them, is the confidence of the th category ; is the object confidence; is the probability of the th category ; is the network prediction value of the th category ; t is the network prediction value; From the confidences of all classes, select the class index corresponding to the maximum value as the final predicted class.

Images