Fine-grained target detection method and system capable of identifying easily-confused student behaviors
By constructing the FG-YOLO framework and combining YOLOv8 and MobileNetv3 models, the problem of easily confused student behaviors in the classroom environment is solved, achieving fast and accurate detection of easily confused student behaviors and improving the target detection performance in smart education scenarios.
Patent Information
- Application Number
- CN202410730554.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2025-12-30
AI Technical Summary
In existing classroom student behavior recognition solutions, traditional deep learning target detection models are easily affected by behavioral confusion in classroom scenarios, especially the difficulty in distinguishing between mobile phone use and reading/writing.
A two-stage detection algorithm is used to construct the FG-YOLO framework, which combines YOLOv8 and MobileNetv3 models. Through the information fusion module MSFM and the bridging mechanism module CSCM, fine-grained target detection of easily confused student behaviors is achieved.
The ability to quickly and accurately detect and differentiate easily confused student behaviors in a classroom setting improves the performance and accuracy of target detection tasks.
Smart Images
Figure CN121236346A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of target detection, and particularly relates to a fine-grained target detection method and system capable of identifying easily confused student behaviors. BACKGROUND
[0002] The recognition of student behaviors is mostly based on the research in the field of computer vision, and the target detection algorithm based on pictures is currently the mainstream scheme for behavior recognition.
[0003] According to the development history of the target detection algorithm, the target detection algorithm can be roughly divided into three categories: traditional target detection algorithm, target detection algorithm based on convolutional neural network (CNN) and newly emerging target detection algorithm based on Transformer in recent years. The target detection algorithm based on CNN can be roughly divided into two categories according to its detection mechanism: two-stage detection algorithm and one-stage detection algorithm. The one-stage detection algorithm based on CNN can be divided into two categories: anchor-based detection and keypoint-based detection.
[0004] The existing classroom student behavior recognition scheme is based on the target detection algorithm YOLO, however, due to the complexity of the classroom scene, including the diversity of student behaviors and the uncertainty of environmental factors, the traditional deep learning target detection model is easily affected by behavior confusion in this scene, thereby reducing the practicability of the model. When using the YOLO target detection framework to detect classroom student behaviors, two behaviors that are often confused are using a mobile phone and reading and writing, the difference between the two behaviors is only that the student holds different objects, so they are extremely easy to confuse. SUMMARY
[0005] The present application aims to overcome the shortcomings of the prior art and proposes a fine-grained target detection method and system capable of identifying easily confused student behaviors, so as to solve the problem that behaviors are easily confused when detecting student behaviors in a classroom environment, thereby improving the performance of the target detection task in the intelligent education scene.
[0006] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows: a fine-grained target detection method capable of identifying easily confused student behaviors, the method comprising the following steps:
[0007] S1, receiving image information;
[0008] S2. Construct and train a fine-grained target detection framework FG-YOLO based on a two-stage detection algorithm. The first stage of FG-YOLO is used to identify easily confused student behaviors and non-confusing student behaviors from image information. The second stage of FG-YOLO is used to perform fine-grained discrimination of the easily confused student behaviors.
[0009] S3. Input the image information into the trained FG-YOLO to achieve fine-grained target detection of easily confused student behaviors.
[0010] Preferably, in step S2, the FG-YOLO first stage includes the YOLOv8 target detection algorithm, which is used to identify easily confused student behaviors and difficult-to-confuse student behaviors from image information.
[0011] Preferably, in step S2, the FG-YOLO first stage includes an information fusion module MSFM, which includes a first upsampling layer, a first concat layer, a second upsampling layer, a second concat layer, and a downsampling layer, wherein the first upsampling layer, the first concat layer, the second upsampling layer, the second concat layer, and the downsampling layer are connected in series; for the different scale features P3, P4, and P5 extracted by the YOLOv8 target detection algorithm, feature P5 is input to the first upsampling layer, feature P4 is input to the first concat layer, feature P3 is input to the second concat layer, and the downsampling layer outputs the fused feature Pm of features P3, P4, and P5.
[0012] Preferably, in step S2, the first stage of FG-YOLO includes an SPP-Net module, which is used to integrate images of different sizes into images of a uniform size before sending them to the second stage of FG-YOLO for refined recognition.
[0013] Preferably, the output feature maps after concatenating the different scale features P3, P4, and P5 extracted by the YOLOv8 target detection algorithm with the feature Pm output by the information fusion module MSFM, and the output feature maps after concatenating the feature Pm output by the information fusion module MSFM with the output features of the SPP-Net module, are sent to the second stage after their sizes are transformed to a uniform size.
[0014] Preferably, a bridging mechanism module CSCM is introduced between the first and second stages of the FG-YOLO to transfer the general target detection features of the first stage to the next stage to complete fine-grained target detection.
[0015] Preferably, the bridging mechanism module CSCM employs a self-attention mechanism and a channel attention mechanism, specifically:
[0016] (1) The features of different scales extracted in the first stage of FG-YOLO are processed through a channel attention mechanism to obtain feature Fc1, that is:
[0017] F c1 =CA(∑P i );
[0018] Among them, P i This represents the different scale features extracted in the first stage of FG-YOLO; CA represents the channel attention mechanism operation.
[0019] (2) Perform a self-attention mechanism operation on feature Fc1 to obtain feature Fs1, that is:
[0020] F s1 =SA*F c1 );
[0021] Where SA represents the self-attention mechanism operation;
[0022] (3) The features Fs1 extracted at different scales in the first stage of FG-YOLO and the features input to the second stage of FG-YOLO are subjected to channel attention mechanism operation to obtain feature Fc2, that is:
[0023] F c2 =CA*∑P i +F s1 +∑B j );
[0024] Among them, B j This represents the features input to the FG-YOLO two-stage process;
[0025] (4) Perform a self-attention mechanism operation on feature Fc2 to obtain feature Fs2. Feature Fs2 is then fed into the FG-YOLO two-stage process to complete fine-grained target detection, i.e.:
[0026] F s2 =SA(F c2 );
[0027] SA represents the self-attention mechanism operation.
[0028] Preferably, the FG-YOLO two-stage model includes the MobileNetv3 model, which is used to perform fine-grained discrimination of easily confused student behaviors.
[0029] Furthermore, this invention also proposes a fine-grained target detection system for identifying easily confused student behaviors based on the above method. The system includes a camera and a computer, wherein the camera is used to capture images of student behaviors; and the computer is used to perform fine-grained target detection of easily confused student behaviors according to the method.
[0030] The technical effects of this invention are as follows: It proposes a fine-grained target detection method and system that can identify easily confused student behaviors. Based on the constructed FG-YOLO framework, it can quickly and accurately detect easily confused student behaviors in the classroom environment and further refine and distinguish them, thus solving the problem of easily confused behaviors when detecting student behaviors in the classroom environment, thereby improving the performance of target detection tasks in smart education scenarios. Attached Figure Description
[0031] Figure 1 This is a simplified schematic diagram of the FG-YOLO fine-grained target detection framework according to an embodiment of the present invention;
[0032] Figure 2 This is a schematic diagram of the one-stage detection algorithm structure according to an embodiment of the present invention;
[0033] Figure 3 This is a schematic diagram of the two-stage detection algorithm structure according to an embodiment of the present invention. Detailed Implementation
[0034] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, in order to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation. The terms "first," "second," etc., used in this invention are for convenience of description only. To make the technical solution of the present invention clearer, the present invention will be explained and illustrated through the following embodiments:
[0035] A fine-grained target detection method for identifying easily confused student behaviors, the method comprising the following steps:
[0036] S1. Receive image information. The image information includes the classroom background and classroom images. The classroom images are extracted from the video stream at a certain frame rate (e.g., 6 frames / second), which can be achieved using common video reading tools such as OpenCV.
[0037] S2. Construct and train a fine-grained target detection framework FG-YOLO based on a two-stage detection algorithm. The first stage of FG-YOLO is used to identify easily confused student behaviors and non-confusing student behaviors from image information. The second stage of FG-YOLO is used to perform fine-grained discrimination of the easily confused student behaviors.
[0038] S3. Input the image information into the trained FG-YOLO to achieve fine-grained target detection of easily confused student behaviors.
[0039] The FG-YOLO constructed in this embodiment is specifically designed for fine-grained target detection tasks in classroom scenarios, specifically for identifying easily confused student behaviors. It can identify easily confused student behaviors and further refine their distinctions, aiming to solve the behavioral ambiguity problem that arises during student behavior recognition in a classroom environment. For example... Figure 1 The diagram shows the overall framework of the FG-YOLO constructed in this embodiment. In the first stage, the FG-YOLO uses the YOLOv8 object detection algorithm as the main detection network to identify easily confused and unconfined student behaviors from image information. The second stage uses the MobileNetv3 model (large type) for fine-grained discrimination of easily confused student behaviors. The YOLOv8 object detection algorithm is existing technology. In this embodiment, it is chosen as the main component of the first stage of FG-YOLO primarily to leverage its excellent object detection performance, thereby improving the accuracy of filtering easily confused and unconfined student behaviors. The MobileNetv3 model is also existing technology. In this embodiment, it is chosen for the second-stage image classification task of FG-YOLO, primarily to leverage its excellent object detection and image classification performance to achieve fine-grained differentiation of easily confused student behaviors.
[0040] The FG-YOLO constructed in this embodiment mainly consists of the following three parts: a two-stage framework for detecting ambiguous student behavior, a bridging mechanism module CSCM that can fully utilize feature information from different scales across multiple stages, and an information fusion module MSFM that can fuse features from different stages to achieve information enhancement goals. The following will combine... Figure 2 and Figure 3 Further explanation of FG-YOLO.
[0041] To differentiate easily confused student behaviors, FG-YOLO's framework operates in two phases: Phase one, based on the YOLOv8 object detection algorithm's feature extractor, captures global contextual information and preliminary target location information, identifying less easily confused categories (e.g., staring at the blackboard, standing, turning around, etc.) and easily confused categories (e.g., playing on a mobile phone and reading / writing). In one possible embodiment, Phase one may also introduce a region proposal mechanism into YOLOv8, requiring a new layer in YOLOv8 for region proposal calculation, and the corresponding training dataset also needs region proposal annotation. Phase two, based on the MobileNetv3 model, performs coarse-to-fine filtering of confused categories.
[0042] To further enhance feature fusion and model performance, the first stage of FG-YOLO in this embodiment includes an Information Fusion Module (MSFM). MSFM fuses features from different dimensions in the first stage and sends them to the second stage. The network in the second stage can fully utilize the fused features to make more accurate judgments when further subdividing obfuscation behaviors. Through MSFM, FG-YOLO can better combine global and local information, improving the accuracy and robustness of object detection tasks.
[0043] Specifically, the information fusion module MSFM in this embodiment includes a first upsampling layer, a first Concat (feature splicing or fusion) layer, a second upsampling layer, a second Concat layer, and a downsampling layer, wherein: the first upsampling layer, the first Concat layer, the second upsampling layer, the second Concat layer, and the downsampling layer are connected in series; for the different scale features P3, P4, and P5 extracted by the YOLOv8 target detection algorithm, feature P5 is input to the first upsampling layer, feature P4 is input to the first Concat layer, feature P3 is input to the second Concat layer, and the downsampling layer outputs the feature Pm after the fusion of features P3, P4, and P5.
[0044] Furthermore, when sending ambiguous student behaviors identified in the first stage to the second-stage network, there is a problem of inconsistent image sizes in the sub-images. This embodiment utilizes the SPP-Net module to integrate images of different sizes into a uniform size before sending them to the second-stage network for refined identification. Compared to the original images, the sub-images sent to the second-stage network retain more subtle feature information of student behaviors, thus enabling better judgment of easily confused behaviors. The SPP-Net module is a commonly used technique by those skilled in the art, and will not be elaborated upon here.
[0045] In summary, in this embodiment, the output feature maps obtained by concatenating the different scale features P3, P4, and P5 extracted by the YOLOv8 object detection algorithm with the features Pm output by the information fusion module MSFM, and the output feature maps obtained by concatenating the features Pm output by the information fusion module MSFM with the output features of the SPP-Net module, are resized to a uniform size before being sent to the second stage. The output feature maps of the above-mentioned FG-YOLO first stage will be sent to the second stage for fine-grained segmentation. Through the detection in the first stage, the student behaviors identified from the image and the accuracy of each behavior can be obtained. Then, it is determined whether the identified student behaviors are easily confused student behaviors. For example, for the acquired student behaviors, it is first determined whether they are easily confused behaviors such as playing on a mobile phone or reading and writing. If so, they are sent to the next stage for further subdivision. Finally, the student's identified behavior and accuracy are given in the second stage; if not, the student behavior and its identification accuracy are directly output.
[0046] Because the input for the second-stage fine-grained identification of easily confused student behaviors is a sub-image cropped from the original image, and the background information of the global image is ignored when identifying the sub-image, a bridging mechanism module (CSCM) is introduced between the first and second stages of FG-YOLO to address this issue and fully utilize the feature information extracted in the first stage. This CSCM is used to pass the general object detection features from the first stage to the next stage to complete fine-grained object detection, thereby improving the identification of easily confused features. The CSCM can be implemented using PyTorch.
[0047] Specifically, the bridging mechanism module CSCM in this embodiment adopts a self-attention mechanism and a channel attention mechanism, and mainly includes the following steps:
[0048] (1) The features of different scales extracted in the first stage of FG-YOLO are processed through a channel attention mechanism to obtain feature Fc1, that is:
[0049] F c1 =CA(∑P i );
[0050] Among them, P i The different scale features extracted in the first stage of FG-YOLO are represented by ; CA represents the channel attention mechanism operation; in this embodiment, i is 3, 4, and 5, that is, the different scale features extracted in the first stage of FG-YOLO are P3, P4, and P5, respectively.
[0051] (2) Perform a self-attention mechanism operation on feature Fc1 to obtain feature Fs1, that is:
[0052] F s1 =SA(Fc1 );
[0053] Where SA represents the self-attention mechanism operation;
[0054] (3) The features Fs1 extracted at different scales in the first stage of FG-YOLO and the features input to the second stage of FG-YOLO are subjected to channel attention mechanism operation to obtain feature Fc2, that is:
[0055] F c2 =CA*∑P i +F s1 +∑B j );
[0056] Among them, B j In this embodiment, B represents the feature input to the FG-YOLO two-stage process. j This is the output of the first stage of FG-YOLO;
[0057] (4) Perform a self-attention mechanism operation on feature Fc2 to obtain feature Fs2. Feature Fs2 is then fed into the FG-YOLO two-stage process to complete fine-grained target detection, i.e.:
[0058] F s2 =SA(F c2 );
[0059] SA represents the self-attention mechanism operation.
[0060] With the help of CSCM, the FG-YOLO two-stage feature extractor receives the semantic features passed from the first stage to obtain more refined student behavior detection results. The FG-YOLO two-stage object detector can better understand and distinguish student behavior, thereby eliminating behavioral ambiguity.
[0061] Furthermore, based on the above method, this invention also proposes a fine-grained target detection system for identifying easily confused student behaviors. The system includes a camera and a computer, wherein the camera is used to capture images of student behaviors; and the computer is used to perform fine-grained target detection of easily confused student behaviors according to the detection method of this invention.
[0062] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.
Claims
1. A fine-grained target detection method that can identify confusing student behaviors, characterized in that: The method comprises the following steps: S1, receiving image information; S2, constructing a fine-grained target detection framework FG-YOLO based on a two-stage detection algorithm and training, wherein the FG-YOLO first stage is used to identify easy-to-confuse student behaviors and not-easy-to-confuse student behaviors from the image information, and the FG-YOLO second stage is used to perform fine-grained discrimination on the easy-to-confuse student behaviors; S3, inputting the image information into the trained FG-YOLO to achieve fine-grained target detection of the easy-to-confuse student behaviors.
2. The method of claim 1, wherein: In the step S2, the FG-YOLO first stage comprises a YOLOv8 target detection algorithm, which is used to identify easy-to-confuse student behaviors and not-easy-to-confuse student behaviors from the image information.
3. The method of claim 2, wherein: In the step S2, the FG-YOLO first stage comprises an information fusion module MSFM, which comprises a first upsampling layer, a first Concat layer, a second upsampling layer, a second Concat layer, and a down-sampling layer, wherein the first upsampling layer, the first Concat layer, the second upsampling layer, the second Concat layer, and the down-sampling layer are connected in series; for different scale features P3, P4, and P5 extracted in the YOLOv8 target detection algorithm, the feature P5 is input into the first upsampling layer, the feature P4 is input into the first Concat layer, and the feature P3 is input into the second Concat layer, and the down-sampling layer outputs a feature Pm fused from the features P3, P4, and P5.
4. The method of claim 2 or 3, wherein the method is characterized by: In the step S2, the FG-YOLO first stage comprises an SPP-Net module, which is used to integrate pictures of different sizes into pictures of a uniform size before sending them to the FG-YOLO second stage for refined identification.
5. The method of claim 4, wherein: The output feature map obtained after the different scale features P3, P4, and P5 extracted in the YOLOv8 target detection algorithm are concatenated with the feature Pm output by the information fusion module MSFM, and the output feature map obtained after the feature Pm output by the information fusion module MSFM is concatenated with the output feature of the SPP-Net module, are sent to the second stage after being transformed to a uniform size.
6. The method of claim 1, wherein: A bridging mechanism module CSCM is introduced between the FG-YOLO first stage and the second stage, which is used to pass the general target detection features of the first stage to the next stage to complete fine-grained target detection.
7. The method of claim 6, wherein the method is a fine-grained object detection method that can identify confusing student behaviors. The bridging mechanism module CSCM adopts a self-attention mechanism and a channel attention mechanism, specifically: (1) the different scale features extracted in the FG-YOLO first stage are operated through a channel attention mechanism to obtain a feature Fc1, i.e. F c1 = CA(∑P i ); wherein P i represents different scale features extracted in the FG-YOLO one-stage; CA represents a channel attention mechanism operation; (2) the feature Fc1 is operated through a self-attention mechanism to obtain a feature Fs1, i.e. F s1 = SA(F c1 ); wherein SA represents a self-attention mechanism operation; (3) the different scale features extracted in the FG-YOLO first stage, the feature Fs1, and the feature input into the FG-YOLO second stage are operated through a channel attention mechanism to obtain a feature Fc2, i.e. F c2 = CA(∑P i + F s1 + ∑B j ); B j represents the input feature of the FG-YOLO two-stage; (4) performing a self-attention mechanism operation on the feature Fc2 to obtain a feature Fs2, and sending the feature Fs2 to the FG-YOLO two-stage to complete fine-grained target detection, that is: F s2 = SA(F c2 ); Wherein, SA represents a self-attention mechanism operation.
8. The method of claim 1, wherein: The FG-YOLO two-stage includes a MobileNetv3 model, which is used for fine-grained discrimination of the confusing student behavior.
9. A fine-grained goal detection system that can identify confusing student behaviors, characterized by: The system includes a camera and a computer, wherein the camera is used to collect student behavior pictures; and the computer is used to perform fine-grained target detection of the confusing student behavior according to the method of any one of claims 1-8.