Classroom small target detection method based on lightweight weighted bidirectional connection multi-detection head

Abstract

The present application relates to the technical field of target detection, and particularly relates to a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head, and proposes a lightweight scheme of a weighted bidirectional connection multi-detection head target detection model, that is, a CBAM attention module is used to solve the problem of student small target behavior detection difficulty in large classroom teaching conditions, while improving the precision and reducing the floating point calculation amount and parameter amount of the model. By placing the CBAM attention mechanism module in the input part of the Backbone module and the Head module of the entire YOLOv5 model, the floating point calculation amount and parameter amount of the model are reduced, the model is more lightweight, and is suitable for lightweight scenes such as classrooms.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head. BACKGROUND

[0002] As the main body of teaching activities, the behavior of students in the classroom teaching process is one of the most direct and important bases for reflecting the learning state of students and the quality of classroom teaching. Schools and teachers can adjust and improve the teaching process according to the real classroom behavior of students, so as to effectively improve the teaching quality. In the traditional method, only the teacher's own observation can be used for judgment, which has great limitations in perspective and direction, and a large amount of manpower and material resources is consumed.

[0003] Based on the deep learning target detection algorithm, the behavior state of students in the classroom is detected and analyzed in the whole period, which breaks the limitations of the traditional method and can judge the situation of each student and the whole class for a longer period of time, and analyze the behavior of students. As one of the important components of artificial intelligence image field, target detection can judge the position, size and category of the target to be recognized in the image or video, which can effectively reduce the participation rate of manual work and greatly improve the teaching efficiency. At present, target detection has been widely used in face recognition of mobile phones and access control, vehicle flow detection and license plate tracking on roads, and industrial detection.

[0004] The existing document "Zhang Dechun, Li Haitao, Li Xun, Zhang Lei. Fishery target detection based on CBAM and BiFPN improved YoLov5 [J]. Fishery modernization, 2022, 49(03): 71-80." increases the CBAM mechanism module in the backbone network, and changes the PANet network structure to BiFPN in the neck to strengthen the multi-scale feature fusion. However, in actual use, although the specific implementation method provided in the document can increase the overall recognition accuracy, it will increase the floating point calculation amount and parameter amount, making the model huge, which cannot be applied to the lightweight scene such as classroom. SUMMARY

[0005] In view of the above problems, the present application provides a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head.

[0006] The technical scheme adopted is a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head, comprising the following steps:

[0007] S1. Load the preset YOLOv5 model and initialize the parameters of the preset YOLOv5 model;

[0008] S2. improving the feature fusion module in the preset YOLOv5 model through a BiFPN feature pyramid;

[0009] S3. adding a feature map detection head in the Neck module of the YOLOv5 model;

[0010] S4. replacing the C3_X component in the Backbone module of the YOLOv5 model with a CBAM attention mechanism module, reducing the floating point calculation amount and parameter amount of the YOLOv5 model, and obtaining an improved YOLOv5 model;

[0011] S5. training the improved YOLOv5 model to obtain a detection model;

[0012] S6. importing the collected student behavior video data in the classroom into the detection model to complete the detection.

[0013] Further, in S2, the layer structure of the Neck module in the YOLOv5 model is changed to a BiFPN feature pyramid for feature fusion, and the multi-scale feature fusion of the feature map is performed through the feature fusion capability of the BiFPN feature pyramid.

[0014] Further, in S3, the feature map detection head includes a CBS component, an Upsample component, a Concat_BiFPN component and a C3_1_F component.

[0015] Optionally, in S3, the C3_1 component in the Backbone module of the YOLOv5 model is output to the Concat_BiFPN component in the feature map detection head.

[0016] Further, in S3, the Neck module in the YOLOv5 model has multiple feature map detection heads, and the CBS component and the C3_1_F component in each feature map detection head have the same parameters as other components with the same name in the Neck module.

[0017] Further, in S3, the size of the feature map detection head is 160x160.

[0018] Optionally, in S4, after the CBAM attention mechanism module is added, the feature map F after passing through the CBS component in the Backbone module is , the compressed features after the feature map F passes through the CBAM attention mechanism module are , and .

[0019] Further, in S4, a CBAM attention mechanism module is added after the C3_X_F component in the output part of the Neck module of the YOLOv5 model, and the detection accuracy of the YOLOv5 model is maintained.

[0020] Optionally, in S4, a CBAM attention mechanism module is added at the input part of the YOLOv5 model Head module to improve the feature fusion capability.

[0021] The application also provides a matching system, a composite contaminated site soil remediation range determination and remediation amount calculation system, which comprises:

[0022] The memory is used for storing the computer program.

[0023] The processor is used for executing the computer program stored in the memory to realize a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head.

[0024] The beneficial effects of the application at least include one of the following;

[0025] 1. The application proposes a lightweight scheme of a weighted bidirectional connection multi-detection head target detection model, that is, a CBAM attention module is used to solve the problem of small target behavior detection in large classroom teaching, which improves the precision and reduces the floating point calculation amount and parameter amount of the model.

[0026] 2. Compared with the existing public literature, the CBAM attention mechanism module is placed in the input part of the Backbone module and the Head module of the entire YOLOv5 model to reduce the floating point calculation amount and parameter amount of the model, so that the model is more lightweight and suitable for lightweight scenes such as classrooms.

[0027] 3. In terms of small target detection precision, the weighted bidirectional connection multi-detection head target detection model can reach 94.6%, which is improved by 9.1% compared with the preset YOLOv5 model.

[0028] 4. In terms of lightweight, the CBAM attention mechanism module is placed in the input part of the Backbone module and the Head module of the entire YOLOv5 model, which reduces the floating point calculation amount and parameter amount of the weighted bidirectional connection multi-detection head target detection model by 13.5% and 22% respectively, so that the model is significantly lightweight. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 It is a preset YOLOv5 model structure diagram;

[0030] Figure 2 It is a weighted bidirectional connection multi-detection head target detection model structure diagram;

[0031] Figure 3 It is a lightweight weighted bidirectional connection multi-detection head target detection model structure diagram;

[0032] Figure 4Loss chart of the weighted bidirectional connection multi-detection head target detection model training set;

[0033] Figure 5 Loss chart of the weighted bidirectional connection multi-detection head target detection model test set;

[0034] Figure 6 Precision and recall chart of the weighted bidirectional connection multi-detection head target detection model;

[0035] Figure 7 mAP chart of the weighted bidirectional connection multi-detection head target detection model;

[0036] Figure 8 Precision chart of each label in the embodiment;

[0037] Figure 9 Floating point calculation and parameter amount chart in the embodiment;

[0038] Figure 10 Experiment chart in the embodiment in terms of Precision and Recall; DETAILED DESCRIPTION

[0039] The present application is further explained by the following specific examples, which should not be construed as limiting the scope of the application. The other advantages and effects of the present application can be easily understood by those skilled in the art from the contents disclosed in the specification. The present application can also be implemented or applied by other different specific embodiments, and the details in the specification can be modified or changed in various ways based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0040] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the components related to the present application in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be arbitrarily changed in terms of shape, number and proportion, and the layout pattern of the components may also be more complex.

[0041] A classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head, comprising the following steps:

[0042] S1. Loading a preset YOLOv5 model and initializing parameters of the preset YOLOv5 model;

[0043] S2. Improving a feature fusion module in the preset YOLOv5 model by a BiFPN feature pyramid;

[0044] S3. Adding a feature map detection head in a Neck module of the YOLOv5 model;

[0045] S4. Replacing the C3_X component in the Backbone module of the YOLOv5 model with the CBAM attention mechanism module to reduce the floating point calculation amount and parameter amount of the YOLOv5 model, and obtaining an improved YOLOv5 model;

[0046] S5. Training the improved YOLOv5 model to obtain a detection model;

[0047] S6. Importing the collected student behavior video data in the classroom into the detection model to complete the detection

[0048] The purpose of such design is to propose a lightweight scheme of a weighted bidirectional connection multi-detection head target detection model, that is, to use the CBAM attention module to solve the problem of small target behavior detection of students in large classrooms during classroom teaching, to improve the accuracy while reducing the floating point calculation amount and parameter amount of the model. By placing the CBAM attention mechanism module in the input part of the Backbone module and the Head module of the entire YOLOv5 model, the floating point calculation amount and parameter amount of the model are reduced, and the model is more lightweight, which is suitable for lightweight scenes such as classrooms.

[0049] In specific implementation, as shown in Figure 3 A classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head, comprising the following steps:

[0050] S1. Loading a preset YOLOv5 model and initializing parameters of the preset YOLOv5 model;

[0051] S2. Improving the feature fusion module in the preset YOLOv5 model by BiFPN feature pyramid, wherein the layer structure of the Neck module in the YOLOv5 model is changed to BiFPN feature pyramid for feature fusion, and the multi-scale feature fusion of the feature map is performed by the feature fusion capability of the BiFPN feature pyramid;

[0052] S3. Adding a feature map detection head to the Neck module of the YOLOv5 model, wherein the feature map detection head comprises a CBS component, an Upsample component, a Concat_BiFPN component and a C3_1_F component, the C3_1 component in the Backbone module of the YOLOv5 model is output to the Concat_BiFPN component in the feature map detection head, the CBS component and the C3_1_F component in each feature map detection head are the same as the parameters of other components with the same name in the Neck module, and the size of the feature map detection head is 160x160;

[0053] S4. Replace the C3_X component in the Backbone module of the YOLOv5 model with a CBAM attention mechanism module to reduce the floating-point calculation amount and parameter amount of the YOLOv5 model, and obtain an improved YOLOv5 model. After the CBAM attention mechanism module is added, the feature map F after passing through the CBS component in the Backbone module is When the feature map F passes through the CBAM attention mechanism module, the compressed features are and The CBAM attention mechanism module is added after the C3_X_F component in the output part of the Neck module of the YOLOv5 model to maintain the detection accuracy of the YOLOv5 model. The CBAM attention mechanism module is added in the input part of the Head module of the YOLOv5 model to improve the feature fusion capability.

[0054] S5. Train the improved YOLOv5 model to obtain a detection model.

[0055] S6. Import the collected student behavior video data in the classroom into the detection model to complete the detection

[0056] In this embodiment, in order to verify the overall effect, multiple groups of verification are performed, wherein Figure 2 and Figure 3 are a weighted bidirectional connection multi-detection head target detection model and a lightweight weighted bidirectional connection multi-detection head target detection model, respectively. The two models are similar in accuracy, and the main difference is that the latter further reduces the floating-point calculation amount and parameter amount.

[0057] As shown in Figures 4 to 7 , the loss curves of the training set and the test set of the weighted bidirectional connection multi-detection head target detection model can be seen that the improved algorithm converges faster and better. At the same time, the AP of each label is very high, and the performance of the improved model is excellent.

[0058] At the same time, the results of the preset YOLOv5 model, the weighted bidirectional connection multi-detection head target detection model and the lightweight weighted bidirectional connection multi-detection head target detection model are compared horizontally as shown in Figure 8 , Figure 9 and Figure 10 .

[0059] Figure 8 Compared with the preset YOLOv5 model, the lightweight weighted bidirectional connection multi-detection head target detection model improves the head_up label by 29% compared with the preset YOLOv5 model, improves the head_down label by 27.8% compared with the preset YOLOv5 model, and improves the head_aside label by 8.3% compared with the preset YOLOv5 model.

[0060] Figure 9 In terms of mAP, the mAP of the lightweight weighted bidirectional connection multi-head target detection model is improved by 1.1% compared with the weighted bidirectional connection multi-head target detection model, and by 9.2% compared with the original YOLOv5 model, which is significant.

[0061] In terms of floating point calculation, the lightweight weighted bidirectional connection multi-head target detection model not only improves the accuracy, but also greatly reduces the floating point calculation, which can be seen that the floating point calculation of the lightweight weighted bidirectional connection multi-head target detection model is only 18 GFLOPs, which is reduced by 13.5% compared with the weighted bidirectional connection multi-head target detection model.

[0062] In terms of parameter amount, only 6.49M, which is reduced by 22% compared with the weighted bidirectional connection multi-head target detection model, and more than 10% compared with the original YOLOv5 model, effectively reducing the demand of the model for video memory

[0063] Figure 10 Like the weighted bidirectional connection multi-head target detection model, the lightweight weighted bidirectional connection multi-head target detection model is slightly lower than the YOLOv5 model in the Precision value; and in terms of Recall, the lightweight weighted bidirectional connection multi-head target detection model is improved by 8% compared with the YOLOv5 model.

[0064] In summary, the lightweight weighted bidirectional connection multi-head target detection model improves the mAP by adding weighted bidirectional connection and large size feature detection head, compared with the preset YOLOv5 model, especially the AP of the head label is obviously improved;

[0065] By adding CBAM attention mechanism, compared with the weighted bidirectional connection multi-head target detection model, not only the model mAP value is slightly improved, but also the floating point calculation and parameter amount are effectively reduced, which reduces the complexity of the model and the demand for video memory, and is more conducive to the popularization of the model.

[0066] In terms of working principle, CBAM attention mechanism module is also called convolution attention module, which is a lightweight module based on VGG structure, which can be seamlessly integrated in the architecture of CNN. CBAM attention mechanism module includes channel and spatial calculation channel attention module which first processes the feature map after input, and the calculated result is calculated by spatial attention mechanism module to output the feature.

[0067] The overall formula is shown below, where Ms and Mc represent the results calculated by the spatial and channel modules respectively, which is used to represent that the elements are multiplied in turn:

[0068]

[0069] In the attention mechanism, the weight distribution distinguishes the degree of attention to different information. The attention mechanism can be regarded as a multi-layer perceptron composed of weighted average, key and query matrix. Given the query Q related to the task, the correlation between different keys K in the source and Q is calculated, that is, the weight coefficient of different value V in the source is calculated. The attention value is the weighted average result of V, and its formula is:

[0070]

[0071] In the first stage, the correlation between Q and K is calculated to obtain the weight of V. There are three methods:

[0072] Dot product method:

[0073]

[0074] Calculate the cosine similarity:

[0075]

[0076] Introduce neural network:

[0077]

[0078] The second stage of calculation is to normalize the output of the first stage, and the i-th value of a is the weight value.

[0079]

[0080] In the third stage, the value and its corresponding weight are multiplied and added to calculate the attention value.

[0081]

[0082] The CBAM attention mechanism module judges the content of the image, distinguishes whether it is meaningful, and then gives greater weight to the meaningful part of the channel attention module. In order to save computing power, the spatial dimension of the feature map after extracting the input feature is compressed, and the effective information in the space is gathered, as follows.

[0083] A1. Compress the feature map F

[0084] The feature map F is obtained after feature extraction by the model. The general compression method is AvagePooling, and GlobalMax Pooling is added in the CBAM attention mechanism to improve the performance of the model. The AvagePooling is average pooling, and the GlobalMax Pooling is global maximum pooling. The two pooling methods are used to calculate more efficient attention values. The feature map F is subjected to different pooling in the H and W dimensions, and finally a feature dimension matrix of 1×1×C is obtained.

[0085] A2. MLP calculation

[0086] The obtained feature dimension matrix is input into the MLP shared network (Multi-Layer Perceptron), which includes multiple perception layers and hidden layers, and the number of neurons in different layers is also different, which is C / r (r is the reduction rate) and C, respectively. The first layer is calculated by the Relu activation function.

[0087] A3. Obtain channel attention feature Mc

[0088] After the feature is calculated by the multi-layer perception network, it is added by the Element-wise operation, and then the Mc attention feature is obtained by the activation function. The calculation formula is as follows:

[0089]

[0090] Compared with the channel attention module in the CBAM attention mechanism module, the spatial attention module focuses on the position of the meaningful part of the image. In the CBAM model, it is a supplement to the channel attention module. The combination of the two models can effectively improve the robustness of the model.

[0091] First, the input feature map is subjected to global pooling and average pooling in the channel dimension to obtain two H×W×1 feature maps, which are used to compress the channel size and are beneficial to the subsequent learning of spatial features.

[0092] Then, the obtained two H×W×1 feature maps are subjected to channel to obtain a feature map with a dimension of H×W×2.

[0093] Finally, a 7×7 convolution kernel is used to reduce the dimension to 1 channel, and then the sigmoid is calculated to obtain the spatial attention feature Ms by multiplying the initial input feature.

[0094] The calculation formula is as follows:

[0095]

[0096] Based on the above, Backbone is used as feature extraction in the target detection model, and then the extracted features are given to the network designed later for calculation operation. The backbone network of the weighted bidirectional connection multi-detection head target detection model is the same as YOLOv5, which is mainly composed of a plurality of CBS, C3_X and SPPF modules.

[0097] The role of the C3_X module in Backbone is to use the advantages of the residual structure in the Bottleneck module to reduce the parameter amount and reduce the network speed while preserving the performance, thereby optimizing the model. However, the parameter amount is still huge.

[0098] The CBAM attention mechanism can pay more attention to key information features and ignore other parts. In classroom student behavior detection, more features of student action areas can be extracted, and background information can be reduced to efficiently use computing resources. Therefore, the CBAM attention mechanism module is used to replace the third and fourth C3_X modules in Backbone, and the structure is as shown in Figure 3 The number in the module represents the number of CBAM modules, such as CBAM_3, which means three CBAM modules are connected in series.

[0099] The CBAM attention mechanism module is divided into channel attention module and spatial attention module. This module can focus more on student behavior information, reduce the amount of calculation and parameter, and the lighter and faster target detection model can be widely used in actual classroom teaching.

[0100] After replacing the CBAM attention mechanism module, the operation process in Backbone network changes. The feature map after passing through the CBS module is F( ), and the compressed features after passing through the channel attention module and the spatial attention module are and , respectively.

[0101] Although replacing the CBAM attention mechanism module in Backbone can reduce the parameter amount of the model and improve the running speed. However, since two C3_X modules are cancelled, important information of the feature map may be ignored by Backbone during model operation, which reduces the mAP of the model. Therefore, in this embodiment, a CBAM attention mechanism module is added after each C3_X_F module in the output part of the Neck layer of the weighted bidirectional connection multi-detection head target detection model, and the input of the Head part is changed to the CBAM module to improve the feature fusion capability of the model. The C3 module in the Neck is different from that in Backbone, mainly that the Bottleneck module cancels the residual structure.

[0102] In this embodiment, a student classroom behavior dataset is constructed for student behavior target detection in a classroom teaching environment. The dataset used in the experiment is constructed by using different classes of different courses in different colleges and universities. After slicing, more than 700 pictures with large differences are selected for labeling. The labels are divided into body labels: stand, sit and sleep, and head labels: look up, look down and side head, so that each student has two labeled boxes. The average number of people in the classroom is about 50, and the number of labeled boxes is about 100.

[0103] In view of the problem that the precision of the YOLOv5 model is not high in small target detection, a new improved model, a weighted bidirectional connection multi-detection head target detection model, is proposed. The improvement of the model is mainly in two aspects: on the one hand, the bidirectional cross-scale connection method and the weighted feature fusion capability of the BiFPN feature pyramid structure are used to improve the Neck layer, and the multi-scale feature fusion capability of the model is enhanced; on the other hand, a 160 The weighted bidirectional connection multi-detection head target detection model can achieve 94.6% after experimental training and verification, which is improved by 9.1% compared with the preset YOLOv5 model.

[0104] In view of the problem that the weighted bidirectional connection multi-detection head target detection model has too high floating point calculation and parameter quantity, a new lightweight improvement method, the CBAM attention mechanism module, is proposed. The lightweight weighted bidirectional connection multi-detection head target detection model uses the characteristics of the CBAM attention mechanism to make the model more focused on the features required by the task, so that the model is more lightweight.

[0105] The CBAM attention mechanism module is mainly used to replace the C3 module in the Backbone structure and is added to the output position of the Neck layer. After experimental training and verification, the mAP of the lightweight weighted bidirectional connection multi-detection head target detection model reaches 95.7%, which is improved by 1.1% compared with the weighted bidirectional connection multi-detection head target detection model. FLOPs (G) and parameter quantity (M) are reduced to 18 GFLOPs and 6.49 M, which are reduced by 13.5% and 22% compared with the floating point calculation and parameter quantity of the weighted bidirectional connection multi-detection head target detection model, so that the model is significantly lightweight.

[0106] In this embodiment, a classroom small target detection system based on a lightweight weighted bidirectional connection multi-detection head is also provided, characterized by comprising:

[0107] a memory for storing a computer program;

[0108] A processor is used for executing a computer program stored in a memory to realize a classroom small target detection method based on a lightweight weighted bidirectional connection multi-detection head.

[0109] Finally, it should be noted that: the above is only the preferred embodiments of the present application, and is not intended to limit the present application, although the foregoing embodiments of the application are described in detail, for those skilled in the art, it still can be modified, or the equivalent replacement of part of the technical features, recorded in the foregoing embodiments of the technical solution, within the spirit and principles of the present application, any modification, equivalent replacement, improvement, etc., should be included within the scope of the present application.

Claims

1. A method for detecting small targets in a classroom based on a lightweight weighted bidirectional connection multi-head, characterized in that, The method comprises the following steps: S1. loading a preset YOLOv5 model and initializing parameters of the preset YOLOv5 model; S2. improving a feature fusion module in the preset YOLOv5 model by using a BiFPN feature pyramid, specifically, changing a layer structure of a Neck module in the YOLOv5 model to the BiFPN feature pyramid for feature fusion, and performing multi-scale feature fusion on feature maps by using a feature fusion capability of the BiFPN feature pyramid; S3. adding a feature map detection head to the Neck module of the YOLOv5 model, wherein the feature map detection head comprises a CBS component, an Upsample component, a Concat_BiFPN component and a C3_1_F component; the C3_1 component in a Backbone module of the YOLOv5 model is output to the Concat_BiFPN component in the feature map detection head, and the Neck module of the YOLOv5 model has multiple feature map detection heads, and the CBS component and the C3_1_F component in each feature map detection head have the same parameters as other components with the same name in the Neck module; S4. replacing a C3_X component in a Backbone module of the YOLOv5 model by using a CBAM attention mechanism module, reducing a floating point calculation amount and a parameter amount of the YOLOv5 model, and obtaining an improved YOLOv5 model; S5. training the improved YOLOv5 model to obtain a detection model; S6. importing collected student behavior video data in a classroom into the detection model to complete detection.

2. The classroom small target detection method based on lightweight weighted bidirectional connection multi-detection head according to claim 1, characterized in that, In S3, the size of the feature map detection head is 160x160.

3. The classroom small target detection method based on lightweight weighted bidirectional connection multi-detection head according to claim 1, characterized in that, In S4, after the CBAM attention mechanism module is added, the feature map F after passing through the CBS component in the Backbone module is When the feature map F passes through the CBAM attention mechanism module, the compressed features are respectively and .

4. The classroom small target detection method based on lightweight weighted bidirectional connection multi-detection head according to claim 3, characterized in that, In S4, a CBAM attention mechanism module is added after a C3_X_F component in an output part of a Neck module of the YOLOv5 model to maintain detection accuracy of the YOLOv5 model.

5. The lightweight weighted bidirectional connection multi-detection head-based classroom small target detection method according to claim 4, characterized in that, In S4, a CBAM attention mechanism module is added in an input part of a Head module of the YOLOv5 model to improve feature fusion capability.

6. A lightweight weighted bidirectional connection multi-detection head-based classroom small target detection system, characterized in that, The method comprises: a memory for storing a computer program; a processor for executing the computer program stored in the memory to implement the classroom small target detection method based on the lightweight weighted bidirectional connection multi-detection head according to any one of claims 1 to 5.

Images