A method and system for classroom student posture recognition based on dual-trend feature fusion

Through the dual-backbone feature fusion method, combined with convolutional neural network and visual Transformer, multi-scale features are extracted and integrated, which solves the efficiency and accuracy problems of multi-objective students' posture recognition in classroom environment, and realizes efficient recognition in complex scenarios. Meets the real-time needs of classroom teaching.

CN118658211BActive Publication Date: 2024-12-03NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411148242.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-03
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

In complex and changeable classroom environments, prior art is difficult to achieve efficient and accurate multi-objective student posture recognition, and is often at the expense of efficiency and cannot meet the demanding requirements of real-time and computing efficiency.

Method used

Using a method based on dual backbone feature fusion, combining CNN and Transformer networks, multi-scale features are extracted through lightweight shuffled backbone networks and sliding window visual backbone networks, and feature integration and prediction are carried out through attention mechanisms and decoupled head prediction structures. Realize accurate recognition of students' postures in classrooms.

Benefits of technology

It realizes accurate and efficient identification of multi-objective student postures in complex scenarios, improves detection accuracy and calculation speed, meets the need for real-time system response in classroom teaching, and ensures the fluency and coherence of the teaching process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658211B_ABST
    Figure CN118658211B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for classroom student posture recognition based on dual-trunk feature fusion, which belongs to the field of computer vision and educational technology. The method includes: obtaining live pictures or videos of students in the classroom; using a dual-trunk feature extraction network to extract features from the live pictures or videos; splicing and fusing the feature maps of the same corresponding scale obtained by each trunk according to the corresponding proportion to obtain three layers of new feature maps; inputting the three layers of new feature maps into a decoupling head prediction structure, performing feature map integration and prediction data decoding calculation to obtain prediction frame information; preprocessing the prediction frame information to obtain a prediction frame and its corresponding confidence, and performing non-maximum suppression on the prediction frame to obtain a classroom student posture recognition result. The present invention effectively mines and fuses multi-scale visual features through a dual-trunk network feature extraction method, and realizes accurate and efficient recognition of multi-target student postures in complex scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision and educational technology, and in particular to a method and system for classroom student posture recognition based on dual-trunk feature fusion. Background Art

[0002] The development trend of smart education has created an urgent need for real-time monitoring and evaluation of classroom teaching. As a basic link, classroom student posture recognition aims to capture the students' engagement state and lay the foundation for subsequent teaching behavior analysis and optimization of teacher-student interaction. Its importance is self-evident. However, achieving efficient and accurate multi-target student posture recognition in a complex and changeable actual classroom environment is still an extremely challenging task.

[0003] First, the classroom scene itself has many disturbance factors, such as light changes, perspective differences, target occlusion, etc., which pose a severe test to the target detection algorithm based on computer vision. Secondly, the classroom teaching process places stringent requirements on the real-time performance and computing efficiency of the system, and the algorithm efficiency needs to be optimized in a targeted manner. Furthermore, since it is necessary to recognize and classify multiple postures of multiple students at the same time, the complexity of the problem is also superior to that of single target detection.

[0004] At present, although some preliminary attempts in this field have made some progress in academia and industry, most of them have defects such as complex models, large amount of calculations, and poor generalization ability, which makes it difficult to truly meet the needs of actual applications. Some studies use attention mechanisms or multi-task learning frameworks to improve detection accuracy, but often at the expense of efficiency. Other works directly apply existing general target detection models to classroom scenarios, but because they are not optimized for the characteristics of the classroom environment, the results are not ideal. Summary of the invention

[0005] In order to overcome the problems existing in the prior art, the present invention provides a method and system for classroom student posture recognition based on dual-trunk feature fusion, which can realize accurate and efficient recognition of multi-target student postures in complex scenarios, laying a foundation for subsequent teaching behavior analysis and optimization of teacher-student interaction. The present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for identifying student postures in a classroom based on dual-trunk feature fusion, comprising:

[0007] Get live pictures or videos of students in the classroom;

[0008] A dual-trunk feature extraction network is used to extract features from live pictures or videos, and each trunk obtains three feature maps of different scales;

[0009] The feature maps of the same scale obtained from each backbone are concatenated and fused according to the corresponding proportions to obtain three layers of new feature maps;

[0010] The three layers of new feature maps are input into the decoupling head prediction structure to perform feature map integration and prediction data decoding calculation to obtain the prediction bounding box information;

[0011] Preprocess the predicted frame information to obtain the predicted frame and its corresponding confidence, and perform non-maximum suppression on the predicted frame to obtain the classroom student posture recognition result;

[0012] The dual-trunk feature extraction network includes a CNN-based lightweight shuffled backbone network and a Transformer-based sliding window visual backbone network;

[0013] The CNN-based lightweight shuffle backbone network includes a shuffle space downsampling unit module and a shuffle basic unit module;

[0014] The Transformer-based sliding window visual backbone network includes a visual segmentation layer, a linear encoding module, a window grouping module A and a window grouping module B;

[0015] The CNN-based lightweight shuffle backbone network and the Transformer-based sliding window visual backbone network are spliced ​​together through the features extracted by each backbone network;

[0016] The operation on the sampling unit module in the shuffle space is:

[0017] S1, the input feature map X1 is first operated by the basic convolution module, the basic convolution module point convolution + batch normalization + ReLu activation function, and then the downsampling depth separable convolution module is operated. The downsampling depth separable convolution module is depth separable convolution + batch normalization operation, and then the basic convolution module is operated again to obtain the output feature map X11;

[0018] S2, first perform downsampling depth-separable convolution module operation on the input feature map X1, and then perform basic convolution module operation to obtain feature map X12;

[0019] S3, concatenate X11 and X12 in the channel dimension to obtain the feature map Y1;

[0020] S4, shuffle the feature map Y1 in the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as ,in, is the feature map obtained by shuffling the feature map Y1 in the channel dimension, group is the number of groups, is the batch size, h and w are the height and width of feature map Y1 respectively, c is the number of channels of feature map Y1, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped;

[0021] The calculation operation in the shuffle basic unit module is:

[0022] S1, the feature map obtained from the shuffled space downsampling unit module First divide the branches into , and divide it into ,in, For the feature map The first branch obtained by branch division is For the feature map The second branch obtained by branch division is The input feature map The number of channels;

[0023] S2, the first branch First, perform the basic convolution module operation, and then perform the depthwise separable convolution module operation. The depthwise separable convolution module is a depthwise separable convolution + batch normalization operation, and then perform the basic convolution module operation to obtain the operation output. ;

[0024] S3, and Concatenate the channel dimensions to get the feature map Y s ;

[0025] S4, for feature map Y s Perform a shuffle operation on the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as , where group is the number of groups, is the batch size, h and w are the feature map height and width respectively, C s is the feature map Y s The number of channels, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped, and the feature map is finally output , is the feature map sequence output by the shuffled basic unit module, including multiple feature maps [Os1, Os2, Os3] for subsequent feature fusion operations;

[0026] The visual segmentation layer transforms the input feature map X1 using convolution operation to obtain the output feature map X1 p , where H and W represent the height and width of the feature map X1 respectively;

[0027] The linear coding module includes a linear coding layer and two sliding window vision modules. The linear coding layer is a feature map X1 obtained after the visual segmentation layer operation. p Follow X1 p’ =X1 p E+E pos Perform a linear projection operation, where E is the learnable weight matrix of the linear projection, E pos It's with X1 p The position-dependent learnable position embedding with D dimensions is transformed into the original X1 through linear projection. p The sequence is mapped to X1 with D dimensions p’ sequence;

[0028] The window grouping module A includes a window segmentation layer and two sliding window vision modules, and the window grouping module B includes a window segmentation layer and six sliding window vision modules;

[0029] The window segmentation layer is based on Generate window sequences of different scales in the following way , and then realize the downsampling operation, that is, the height and width of the feature map are halved and the channels are doubled, where is the convolution kernel of the current operation scale, is the cross-layer convolution kernel of the current operation scale, LN is the layer normalization operation, k is the accumulated variable from 0 to i-1, and n is the total number of layers of window segmentation;

[0030] The sliding window vision module includes a serially connected window attention module and a shifted window attention module;

[0031] The operation on the window attention module is:

[0032] S1, for the input sequence Perform layer normalization to provide data distribution for subsequent operations;

[0033] S2, the normalized sequence Divide into M windows , the standard multi-head self-attention is calculated in each window, which is calculated as:

[0034] ;

[0035] in, is a linear projection, calculating the output attention feature map ; W q , W k , W v is a learnable weight matrix, which is used to generate query vector, key vector and value vector respectively; T represents transposition, represents the scaling factor, and B represents the relative position bias;

[0036] S3, and Perform residual connection to obtain ’ ;

[0037] S4, yes ’ Normalize it and get ' ;

[0038] S5, ' Input feedforward network, according to X 1F =σ( 'W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained 1F ;

[0039] S6, ' With X 1F Perform residual connection to get X 1F ’ , as the input of the next module;

[0040] The shift window attention module expands step S2 of the window attention module and introduces a window shift operation based on the window attention module. The specific operation is:

[0041] S1, for the input sequence X 1F ’ Perform layer normalization to provide data distribution for subsequent operations;

[0042] S2, the normalized sequence X 1F ’ First perform the shift operation to generate the shifted version X 1F ’ _s; then the shifted version of X 1F ’ _s is processed according to the calculation method of the window attention layer in step S2 of the window attention module to calculate the output attention feature map _s; the last _s performs reverse shift operation and restores to ;

[0043] S3, and Perform residual connection to obtain ’ ;

[0044] S4, yes ’ Normalize it and get ';

[0045] S5, ' Input feedforward network, according to X 1FF =σ( ' W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained. 1FF ;

[0046] S6, ' With X 1FF Residual connection is performed to obtain the feature map Ow. The final output feature map Ow is a feature map sequence output by the sliding window visual backbone network, including multiple feature maps [Ow1, Ow2, Ow3], which are used for subsequent feature fusion operations.

[0047] Optionally, the feature maps of the same scale obtained from each backbone are concatenated and fused to obtain three layers of new feature maps, including:

[0048] Obtain feature maps Ow1, Ow2, and Ow3 of three scales from the sliding window visual backbone network, a total of three feature maps,

[0049] The feature maps of the same scale are concatenated respectively, i.e. Concat (Os1, Ow1), Concat (Os2, Ow2), Concat (Os3, Ow3), to obtain feature maps O1, O2, O3. The feature maps O1, O2, O3 are passed through the attention multi-layer residual convolution module to obtain feature maps P1, P2, P3;

[0050] Among them, the specific operations on the attention multi-layer residual convolution module are:

[0051] After a basic convolution block, the basic convolution block includes a convolution layer + batch normalization layer + SiLU activation function;

[0052] Use the chunk function to split the output into two vectors evenly; input the second half of the vector into the attention multi-layer residual convolution group, which contains n attention residual convolution units, and append the output of each attention residual convolution unit to the result;

[0053] After a basic convolution block output, the attention residual convolution unit consists of two basic convolution blocks connected in series with the polarized self-attention mechanism, and residual fusion is performed with the input.

[0054] Optionally, the three-layer new feature map is input into the decoupling head prediction structure, and feature map integration and prediction data decoding calculation are performed to obtain prediction border information, including:

[0055] The feature maps P1, P2, and P3 are input into a decoupling head prediction structure, which includes a detection head branch and a classification head branch; both branches first pass through a basic convolution block, and then the detection head obtains a regression branch with a maximum number of channels of 4x16 or the number of channels divided by 4 through a convolution layer; the classification head obtains a classification branch with a number of channels equal to the number of classification categories through a convolution layer; the final output channel number is the sum of the channel numbers of the two branches, and the results of multiple scale feature maps are integrated together to obtain feature prediction data;

[0056] The feature prediction data is split according to the detection and classification information into classification prediction feature data and position prediction feature data, corresponding to the specifications of batch size × sum of feature map size × number of classification head channels and batch size × sum of feature map size × number of detection head channels respectively;

[0057] Calculate the scaling factor of the corresponding scale feature map, the scaling factor is the result of dividing the feature map height by the input image height, take each pixel in the feature map as an anchor point, and take the center coordinates of the pixel point as the coordinates of the center of the anchor point; discretize the classification prediction feature data through Softmax, calculate the probability value of the distance of each anchor point center point to the left, top, right, and bottom of the feature map center point, and obtain l, t, r, and b after weighted summation, which respectively represent the distance values ​​of each anchor point center point to the left, top, right, and bottom of the feature map center point; the coordinates of the feature map center point are (x, y) and calculate the upper left corner coordinates of the predicted bounding box in the feature map and the lower right corner coordinates , the specific calculation is:

[0058] ;

[0059] ;

[0060] The position prediction feature data is decoded into batch size × sum of feature map size × 4, and the predicted bounding box information is obtained by combining the processed classification prediction feature data and position prediction feature data.

[0061] Optionally, preprocessing the predicted frame information to obtain the predicted frame and its corresponding confidence level includes:

[0062] The position prediction feature data, center coordinates and scaling factor in the prediction frame information are multiplied to convert the position prediction feature data from the normalized scale to the input image scale to obtain the position information; the classification prediction feature data is normalized by the Sigmoid function to obtain the maximum probability category and probability value; the position information and the maximum probability category and probability value are integrated to obtain the prediction frame and the final confidence.

[0063] Optionally, the prediction box is subjected to non-maximum suppression to obtain a classroom student posture recognition result, including:

[0064] Arrange the prediction boxes from large to small according to their confidence to form a candidate list;

[0065] Select the prediction box with the largest confidence and put it into the output list, and delete the selected prediction box from the candidate list;

[0066] Calculate the intersection-and-union ratio of the current prediction box and the candidate boxes in the candidate list, and delete the candidate boxes whose intersection-and-union ratio is greater than the set threshold from the candidate list;

[0067] Repeatedly delete the prediction box with the highest confidence, and delete the candidate boxes whose intersection-over-union ratio is greater than the set threshold, until the candidate list is completely cleared;

[0068] Get the final output list;

[0069] The predicted boxes in the final output list are used as the final classroom student posture recognition results.

[0070] In a second aspect, the present invention provides a classroom student posture recognition system based on dual-trunk feature fusion, comprising:

[0071] The acquisition module is used to obtain live pictures or videos of students in the classroom;

[0072] A feature extraction module is used to extract features from live pictures or videos using a dual-trunk feature extraction network, where each trunk obtains three feature maps of different scales;

[0073] The feature fusion module is used to concatenate and fuse the feature maps of the same scale obtained from each backbone according to the corresponding proportion to obtain three layers of new feature maps;

[0074] A prediction information acquisition module is used to input the three-layer new feature map into the decoupling head prediction structure, perform feature map integration and prediction data decoding calculation, and obtain prediction bounding box information;

[0075] The posture recognition module is used to pre-process the predicted frame information to obtain the predicted frame and its corresponding confidence, and to perform non-maximum suppression on the predicted frame to obtain the classroom student posture recognition result;

[0076] The dual-trunk feature extraction network includes a CNN-based lightweight shuffled backbone network and a Transformer-based sliding window visual backbone network;

[0077] The CNN-based lightweight shuffle backbone network includes a shuffle space downsampling unit module and a shuffle basic unit module;

[0078] The Transformer-based sliding window visual backbone network includes a visual segmentation layer, a linear encoding module, a window grouping module A and a window grouping module B;

[0079] The CNN-based lightweight shuffle backbone network and the Transformer-based sliding window visual backbone network are spliced ​​together through the features extracted by each backbone network;

[0080] The operation on the sampling unit module in the shuffle space is:

[0081] S1, the input feature map X1 is first operated by the basic convolution module, the basic convolution module point convolution + batch normalization + ReLu activation function, and then the downsampling depth separable convolution module is operated. The downsampling depth separable convolution module is depth separable convolution + batch normalization operation, and then the basic convolution module is operated again to obtain the output feature map X11;

[0082] S2, first perform downsampling depth-separable convolution module operation on the input feature map X1, and then perform basic convolution module operation to obtain feature map X12;

[0083] S3, concatenate X11 and X12 in the channel dimension to obtain the feature map Y1;

[0084] S4, shuffle the feature map Y1 in the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as ,in, is the feature map obtained by shuffling the feature map Y1 in the channel dimension, group is the number of groups, is the batch size, h and w are the height and width of feature map Y1 respectively, c is the number of channels of feature map Y1, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped;

[0085] The calculation operation in the shuffle basic unit module is:

[0086] S1, the feature map obtained from the shuffled space downsampling unit module First divide the branches into , and divide it into , in, For the feature map The first branch obtained by branch division is For the feature map The second branch obtained by branch division is The input feature map The number of channels;

[0087] S2, the first branch First, perform the basic convolution module operation, and then perform the depthwise separable convolution module operation. The depthwise separable convolution module is a depthwise separable convolution + batch normalization operation, and then perform the basic convolution module operation to obtain the operation output. ;

[0088] S3, and Concatenate the channel dimensions to get the feature map Y s ;

[0089] S4, for feature map Y s Perform a shuffle operation on the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as , where group is the number of groups, is the batch size, h and w are the feature map height and width respectively, C s is the feature map Y s The number of channels, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped, and the feature map is finally output , is the feature map sequence output by the shuffled basic unit module, including multiple feature maps [Os1, Os2, Os3] for subsequent feature fusion operations;

[0090] The visual segmentation layer transforms the input feature map X1 using convolution operation to obtain the output feature map X1 p , where H and W represent the height and width of the feature map X1 respectively;

[0091] The linear coding module includes a linear coding layer and two sliding window vision modules. The linear coding layer is a feature map X1 obtained after the visual segmentation layer operation. p Follow X1 p’ =X1 p E+E pos Perform a linear projection operation, where E is the learnable weight matrix of the linear projection, E pos It's with X1 p The position-dependent learnable position embedding with D dimensions is transformed into the original X1 through linear projection. p The sequence is mapped to X1 with D dimensions p’ sequence;

[0092] The window grouping module A includes a window segmentation layer and two sliding window vision modules, and the window grouping module B includes a window segmentation layer and six sliding window vision modules;

[0093] The window segmentation layer is based on Generate window sequences of different scales in the following way , and then realize the downsampling operation, that is, the height and width of the feature map are halved and the channels are doubled, where is the convolution kernel of the current operation scale, is the cross-layer convolution kernel of the current operation scale, LN is the layer normalization operation, k is the accumulated variable from 0 to i-1, and n is the total number of layers of window segmentation;

[0094] The sliding window vision module includes a serially connected window attention module and a shifted window attention module;

[0095] The operation on the window attention module is:

[0096] S1, for the input sequence Perform layer normalization to provide data distribution for subsequent operations;

[0097] S2, the normalized sequence Divide into M windows , the standard multi-head self-attention is calculated in each window, which is calculated as:

[0098] ;

[0099] in, is a linear projection, calculating the output attention feature map ; W q , W k , W v is a learnable weight matrix, which is used to generate query vector, key vector and value vector respectively; T represents transposition, represents the scaling factor, and B represents the relative position bias;

[0100] S3, and Perform residual connection to obtain ’ ;

[0101] S4, yes ’ Normalize it and get ' ;

[0102] S5, ' Input feedforward network, according to X 1F =σ( 'W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained 1F ;

[0103] The shift window attention module expands step S2 of the window attention module and introduces a window shift operation based on the window attention module. The specific operation is:

[0104] S1, for the input sequence X 1F ’ Perform layer normalization to provide data distribution for subsequent operations;

[0105] S2, the normalized sequence X 1F ’ First perform the shift operation to generate the shifted version X 1F ’ _s; then the shifted version of X 1F ’ _s is processed according to the calculation method of the window attention layer in step S2 of the window attention module to calculate the output attention feature map _s; the last _s performs reverse shift operation and restores to ;

[0106] S3, and Perform residual connection to obtain ’ ;

[0107] S4, yes ’ Normalize it and get ';

[0108] S5, ' Input feedforward network, according to X 1FF =σ( ' W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained. 1FF ;

[0109] S6, ' With X 1FF Residual connection is performed to obtain the feature map Ow. The final output feature map Ow is a feature map sequence output by the sliding window visual backbone network, including multiple feature maps [Ow1, Ow2, Ow3], which are used for subsequent feature fusion operations.

[0110] In a third aspect, the present invention provides a classroom student gesture recognition device, comprising:

[0111] Memory and processor;

[0112] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the classroom student posture recognition method based on dual-trunk feature fusion as described in the first aspect are implemented.

[0113] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for classroom student posture recognition based on dual-trunk feature fusion as described in the first aspect.

[0114] Compared with the prior art, the present invention has the following beneficial effects:

[0115] Aiming at actual classroom application scenarios, the present invention innovatively proposes an efficient classroom student posture recognition method based on dual-trunk feature fusion. The method cleverly combines the advantages of two mainstream network architectures, convolutional neural network and visual Transformer, effectively mines and fuses multi-scale visual features, and realizes accurate and efficient recognition of multi-target student postures in complex scenarios. After the present invention introduces the attention mechanism and lightweight module optimization, the algorithm model not only has high detection accuracy, but also has extremely fast calculation speed and easy deployment, which meets the urgent demand of classroom teaching for real-time response of the system and ensures the fluency and continuity of the teaching process. It lays a technical foundation for improving the quality of classroom teaching and has important theoretical significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 It is an overall flow chart of the method for identifying postures of students in a classroom based on dual-trunk feature fusion of the present invention;

[0117] Figure 2 It is a schematic diagram of the overall network structure of the method of the present invention;

[0118] Figure 3 A schematic diagram of a shuffled space downsampling unit module used in the method of the present invention;

[0119] Figure 4 A schematic diagram of a shuffling basic unit module used in the method of the present invention;

[0120] Figure 5 A schematic diagram of a sliding window vision module used in the method of the present invention;

[0121] Figure 6 Schematic diagram of the attention multi-layer residual convolution module used in the method of the present invention;

[0122] Figure 7 A schematic diagram of the decoupling head prediction structure used in the method of the present invention;

[0123] Figure 8 This is the overall framework diagram of the classroom student posture recognition system based on dual-trunk feature fusion of the present invention. DETAILED DESCRIPTION

[0124] Example 1

[0125] This embodiment introduces a method for classroom student posture recognition based on dual-trend feature fusion, combined with Figure 1 ,include:

[0126] Obtain live pictures or videos of students in the classroom, and pre-process the obtained pictures or videos;

[0127] A dual-trunk feature extraction network is used to extract features from live pictures or videos, and each trunk obtains three feature maps of different scales;

[0128] The feature maps of the same scale obtained from each backbone are concatenated and fused according to the corresponding proportions to obtain three layers of new feature maps;

[0129] The three layers of new feature maps are input into the decoupling head prediction structure to perform feature map integration and prediction data decoding calculation to obtain the prediction bounding box information;

[0130] The predicted frame information is preprocessed to obtain the predicted frame and its corresponding confidence, and the predicted frame is subjected to non-maximum suppression to obtain the classroom student posture recognition result.

[0131] The method of this embodiment mainly involves the following contents.

[0132] 1. Get the image

[0133] Use cameras, mobile phones, cameras and other shooting hardware to obtain live pictures or videos of students in the classroom. Preprocess the obtained pictures, unify the data format, and adjust its size to 1280×1280 pixels. Video data is captured at fixed intervals according to the settings. Data preprocessing unifies the collected data in terms of encoding format, size, scaling, etc., eliminates unfavorable factors for subsequent algorithm processing, and lays the foundation for subsequent feature extraction and posture recognition.

[0134] 2. Dual-backbone feature extraction network feature extraction

[0135] In order to obtain more effective features and obtain more accurate recognition results in subsequent classroom student posture recognition, this embodiment uses a dual-trunk feature extraction network to extract features from the acquired image or video. The dual-trunk feature extraction network structure is as follows: Figure 2 As shown, feature extraction is performed on the on-site pictures or videos, and each backbone network obtains three feature maps of different scales, and then the feature maps of the same scale are fused and spliced ​​together.

[0136] Specifically, the dual-trunk feature extraction network consists of a CNN-based lightweight shuffled backbone network, a Transformer-based sliding window visual backbone network, and an attention multi-layer residual convolution module. The lightweight shuffled backbone network and the sliding window visual backbone network are connected in parallel, and the features extracted by their respective backbone networks are spliced ​​together, and the spliced ​​features are then fused by the attention multi-layer residual convolution module.

[0137] 1.1 Lightweight Shuffle Backbone Network

[0138] The lightweight shuffle backbone network is composed of 1 shuffle space downsampling unit module, 3 shuffle basic unit modules, 1 shuffle space downsampling unit module, 7 shuffle basic unit modules, 1 shuffle space downsampling unit module and 3 shuffle basic unit modules connected in series, and the shuffle basic unit modules are also connected in series.

[0139] Among them, the network structure diagram of the shuffle space downsampling unit module is as follows Figure 3 As shown in FIG. 1 , the module includes two branches. In the first branch, one end of the input is connected to the input of the basic convolution module, the output of the basic convolution module is connected to the input of the downsampling depthwise separable convolution module, and the output of the downsampling depthwise separable convolution module is connected to the input of the basic convolution module; in the other branch, the input of the module is connected to the input of the downsampling depthwise separable convolution module, and the output of the downsampling depthwise separable convolution module is connected to the input of the basic convolution module; the feature maps outputted from the output of the basic convolution modules of the two branches are spliced ​​in the channel dimension splicing module, and then shuffled in the channel dimension shuffling module to obtain the required feature maps.

[0140] The network structure diagram of the shuffle basic unit module is as follows Figure 4 As shown, the input feature map is first divided into branches. In the first branch, one end of the input is connected to the input of the basic convolution module, the output of the basic convolution module is connected to the input of the depthwise separable convolution module, the output of the depthwise separable convolution module is connected to the input of the basic convolution module, and the output of the basic convolution module is connected to the input of the channel dimension splicing module; in the other branch, one end of the input is directly connected to the input of the channel dimension splicing module; the output of the channel dimension splicing module is connected to the input of the channel dimension shuffling module.

[0141] 1.1.1 The operation on the sampling unit module in the shuffle space is:

[0142] S1, the input feature map X1 is first operated by the basic convolution module, the basic convolution module is a point convolution with a convolution kernel of 1×1 and a step size of 1 + batch normalization + ReLu activation function, and then the downsampling depth separable convolution module is operated, the downsampling depth separable convolution module is a depth separable convolution with a convolution kernel of 3×3 and a step size of 2 + batch normalization operation, and then the basic convolution module is operated again to obtain the output feature map X11;

[0143] S2, first perform downsampling depth-separable convolution module operation on the input feature map X1, and then perform basic convolution module operation to obtain feature map X12;

[0144] S3, concatenate X11 and X12 in the channel dimension to obtain the feature map Y1;

[0145] S4, shuffle the feature map Y1 in the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as ,in, is the feature map obtained by shuffling the feature map Y1 in the channel dimension, group is the number of groups, is the batch size, h and w are the height and width of feature map Y1 respectively, c is the number of channels of feature map Y1, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and Swap the positions of the dimensions.

[0146] 1.1.2 The operation in the shuffle basic unit module is:

[0147] S1, the feature map obtained from the shuffled space downsampling unit module First divide the branches into , and divide it into two categories according to the channel dimension Divided into two branches, For the feature map The first branch obtained by branch division is For the feature map The second branch obtained by branch division, c ’ The input feature map The number of channels;

[0148] S2, the first branch First, perform the basic convolution module operation, and then perform the depthwise separable convolution module operation. The depthwise separable convolution module is a depthwise separable convolution with a convolution kernel of 3×3 and a step size of 1 + batch normalization operation. Then perform the basic convolution module operation again to obtain the operation output ;

[0149] S3, and Concatenate the channel dimensions to get the feature map Y s ;

[0150] S4, for feature map Y s Perform a shuffle operation on the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as , where group is the number of groups, is the batch size, h and w are the feature map height and width respectively, C s is the feature map Y sThe number of channels, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped, and the feature map is finally output , is the feature map sequence output by the shuffled basic unit module, including multiple feature maps [Os1, Os2, Os3] for subsequent feature fusion operations.

[0151] 1.2 Sliding Window Vision Backbone Network

[0152] The sliding window visual backbone network is composed of a visual segmentation layer, a linear encoding module, a window grouping module A, a window grouping module B and a window grouping module A in series. The network structure is shown in the figure below. Figure 2 shown.

[0153] 1.2.1 Visual Segmentation Layer

[0154] The visual segmentation layer transforms the input feature map X1 using a convolution operation with a convolution kernel size of 4×4 and a step size of 4, converting the feature map size from (H, W, 3) to (W / 4, H / 4, 48), and obtains the output feature map X1 p , where H and W represent the height and width of the feature map X1 respectively;

[0155] 1.2.2 Linear Encoding Module

[0156] The linear coding module consists of a linear coding layer and two sliding window vision modules. The linear coding layer is the feature map X1 obtained after the visual segmentation layer operation. p Follow X1 p’ =X1 p E+E pos Perform a linear projection operation, where E is the learnable weight matrix of the linear projection, E pos It's with X1 p The position-dependent learnable position embedding with D dimensions is transformed into the original X1 through linear projection. p The sequence is mapped to X1 with D dimensions p’ Sequence, D in this embodiment is 96.

[0157] 1.2.3 Window Grouping Module A and Window Grouping Module B

[0158] The window grouping module A and the window grouping module B are composed of a window segmentation layer and a sliding window vision module in the ratio of 1:2 and 1:6 respectively.

[0159] The window segmentation layer is based on Generate window sequences of different scales in the following way , and then realize the downsampling operation, that is, the height and width of the feature map are halved and the channels are doubled, where is the convolution kernel of the current operation scale, is the cross-layer convolution kernel of the current operation scale, LN is the layer normalization operation, k is the accumulated variable from 0 to i-1, and n is the total number of layers of window segmentation;

[0160] The sliding window vision module is composed of a window attention module and a shift window attention module in series. The specific structure is as follows: Figure 5 As shown,

[0161] Among them, the operation on the window attention module is:

[0162] S1, for the input sequence Perform layer normalization to provide data distribution for subsequent operations;

[0163] S2, the normalized sequence Divide into M windows , the standard multi-head self-attention is calculated in each window, which is calculated as:

[0164] ;

[0165] in, is a linear projection, calculating the output attention feature map ; W q , W k , W v is a learnable weight matrix, which is used to generate query vector, key vector and value vector respectively; T represents transposition, represents the scaling factor, and B represents the relative position bias;

[0166] S3, and Perform residual connection to obtain ’ ;

[0167] S4, yes ’ Normalize it and get ' ;

[0168] S5, ' Input feedforward network, according to X 1F =σ( 'W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained 1F ;

[0169] S6, ' With X 1F Perform residual connection to get X 1F ’ , as the input of the next module;

[0170] The shift window attention module expands step S2 of the window attention module and introduces a window shift operation based on the window attention module. The specific operation is:

[0171] S1, for the input sequence X 1F ’ Perform layer normalization to provide data distribution for subsequent operations;

[0172] S2, the normalized sequence X 1F ’ First perform the shift operation to generate the shifted version X 1F ’ _s; then the shifted version of X 1F ’ _s is processed according to the calculation method of the window attention layer in step S2 of the window attention module to calculate the output attention feature map _s; the last _s performs reverse shift operation and restores to ;

[0173] S3, and Perform residual connection to obtain ’ ;

[0174] S4, yes ’ Normalize it and get ';

[0175] S5, ' Input feedforward network, according to X 1FF =σ( ' W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained. 1FF ;

[0176] S6, ' With X 1FFResidual connection is performed to obtain the feature map Ow. The final output feature map Ow is a feature map sequence output by the sliding window visual backbone network, including multiple feature maps [Ow1, Ow2, Ow3], which are used for subsequent feature fusion operations.

[0177] 3. Feature Fusion

[0178] Feature maps Os1, Os2, Os3, Ow1, Ow2, and Ow3 with scales of 80×80, 40×40, and 20×20 are obtained from the lightweight shuffle backbone network and the sliding window vision backbone network, respectively. There are 6 feature maps in total. The feature maps of the same scale are concatenated accordingly, i.e., Concat (Os1, Ow1), Concat (Os2, Ow2), and Concat (Os3, Ow3) to obtain feature maps O1, O2, and O3. The feature maps O1, O2, and O3 are passed through the attention multi-layer residual convolution module to obtain feature maps P1, P2, and P3.

[0179] Among them, the structure diagram of the attention multi-layer residual convolution module is as follows Figure 6 As shown in the figure, the module consists of a basic convolution block, a split function module, n attention residual convolution units connected in series, and a basic convolution block connected in series. The specific operations on this module are:

[0180] After a basic convolution block with a convolution kernel size of 1x1 and a stride of 1, the basic convolution block consists of a convolution layer + batch normalization layer + SiLU activation function;

[0181] Use the chunk function to split the output into two vectors evenly; input the second half of the vector into the attention multi-layer residual convolution group, which contains n attention residual convolution units, and append the output of each attention residual convolution unit to the result;

[0182] After a basic convolution block output with a convolution kernel size of 1x1 and a stride of 1, the attention residual convolution unit is composed of two basic convolution blocks of size 3x3 and a stride of 1 connected in series with the polarized self-attention mechanism, and residual fused with the input.

[0183] 4. Feature map integration and prediction data decoding calculation

[0184] The feature maps P1, P2, and P3 are input into the decoupling head prediction structure. The network structure diagram is as follows Figure 7As shown in the figure, the decoupling head prediction structure includes a detection head branch and a classification head branch; both branches first pass through a basic convolution block with a convolution kernel size of 3x3 and a stride of 1, and then the detection head passes through a convolution layer with a convolution kernel size of 1x1 and a stride of 1 to obtain a regression branch with a maximum of 4x16 channels or the number of channels divided by 4; the classification head passes through a convolution layer with a convolution kernel size of 1x1 and a stride of 1 to obtain a classification branch with a number of channels equal to the number of classification categories; the final output channel number is the sum of the channel numbers of the two branches, and the results of multiple scale feature maps are integrated together to obtain feature prediction data;

[0185] The feature prediction data is split according to the detection and classification information into classification prediction feature data and position prediction feature data, corresponding to the specifications of batch size × 8400 × number of classification head channels and batch size × 8400 × number of detection head channels respectively;

[0186] Calculate the scaling factor of the corresponding scale feature map, the scaling factor is the result of dividing the feature map height by the input image height, take each pixel in the feature map as an anchor point, and take the center coordinates of the pixel point as the coordinates of the center of the anchor point; discretize the classification prediction feature data through Softmax, calculate the 16 probability values ​​of the distance from each anchor point center point to the left, top, right, and bottom of the feature map center point from 0 to 15, and perform weighted summation to obtain l, t, r, b, which respectively represent the distance values ​​of each anchor point center point from the left, top, right, and bottom of the feature map center point; the coordinates of the feature map center point are (x, y) and calculate the upper left corner coordinates of the predicted bounding box in the feature map and the lower right corner coordinates , the specific calculation is:

[0187] ;

[0188] ;

[0189] The position prediction feature data is decoded into a batch size of ×8400×4, and the predicted bounding box information is obtained by combining the processed classification prediction feature data and position prediction feature data.

[0190] 5. Classroom Student Posture Recognition

[0191] The position prediction feature data, center coordinates and zoom factor in the prediction frame information are multiplied to convert the position prediction feature data from the normalized scale to the input image scale to obtain the position information; the classification prediction feature data is normalized by the Sigmoid function to obtain the maximum probability category and probability value; the position information and the maximum probability category and probability value are integrated to obtain the prediction box and the final confidence. The prediction box is subjected to non-maximum suppression to obtain the classroom student posture recognition result.

[0192] Non-maximum suppression includes:

[0193] Arrange the prediction boxes from large to small according to their confidence to form a candidate list;

[0194] Select the prediction box with the largest confidence and put it into the output list, and delete the selected prediction box from the candidate list;

[0195] Calculate the intersection-and-union ratio of the current prediction box and the candidate boxes in the candidate list, and delete the candidate boxes whose intersection-and-union ratio is greater than the set threshold from the candidate list;

[0196] Repeatedly delete the prediction box with the highest confidence and the candidate box with an intersection-over-union ratio greater than the set threshold until the candidate list is completely cleared;

[0197] Get the final output list;

[0198] The predicted boxes in the final output list are used as the final classroom student posture recognition results.

[0199] The final results of classroom student posture recognition can be used to analyze classroom effectiveness and real-time abnormal posture analysis to guide the optimization of teaching methods and promote the improvement of teaching quality.

[0200] 6. Experiment

[0201] The classroom student posture recognition algorithm model based on dual-trunk feature fusion was compared with the college student classroom posture dataset SCB-Dataset3. The SCB-Dataset3-S dataset includes 5015 pictures and 25810 label data, of which the number of labels for raising hands, reading, and writing are 11207, 10841, and 3762 respectively. The SCB-Dataset3-U dataset includes 671 pictures and 19768 label data, of which the number of labels for raising hands, reading, writing, using mobile phones, mopping heads, and lying on the table are 6, 7826, 2984, 6976, 947, and 1029 respectively. The experimental hardware device operating system is Windows, the CPU is Intel i7-10700F 2.90 GHz, the memory is 16G, and the GPU is NVIDIA GeForce RTX 2060 SUPER. The Python version is 3.9.13, and the deep learning framework uses PyTorch version 1.10.0. The image size we input is 1280*1280. The initial batch size is set to 8, and when the model size increases, it is adjusted according to the GPU performance. The AdamW optimizer with an initial learning rate of lr=1e-3 is used. The number of iterations is 200, and the number of waiting times is 40. That is, if there is no better training indicator in 40 iterations, the experiment is stopped.

[0202] We compared the present invention with the YOLO series algorithms. The results are shown in Table 1. mAP is the mean average precision, which is used to measure the detection accuracy of the algorithm. mAP@50 represents the precision when the intersection-over-union ratio threshold is 0.5, and mAP@50:95 represents the average precision when the intersection-over-union ratio threshold is 0.5 to 0.95.

[0203] Table 1 Recognition results of the method of the embodiment of the present invention and other public methods on SCB-Dataset3

[0204]

[0205] From the experimental results, the algorithm of the present invention has achieved the design goals and algorithm requirements of lightweight, high performance and excellent effect. Overall, under the conditions of consistent model scale, equivalent parameter quantity and operation amount, the present invention has shown a more superior recognition effect than the current mainstream advantage algorithms, which further verifies the innovation, superiority and practical significance of the present invention's classroom student posture recognition method based on dual-trunk feature fusion.

[0206] Example 2

[0207] Based on the same inventive concept as in Example 1, Figure 8 As shown, this embodiment introduces a classroom student posture recognition system based on dual-trunk feature fusion, including:

[0208] An acquisition module is used to acquire live pictures or videos of students in the classroom and pre-process the acquired pictures or videos;

[0209] A feature extraction module is used to extract features from live pictures or videos using a dual-trunk feature extraction network, where each trunk obtains three feature maps of different scales;

[0210] The feature fusion module is used to concatenate and fuse the feature maps of the same scale obtained from each backbone according to the corresponding proportion to obtain three layers of new feature maps;

[0211] A prediction information acquisition module is used to input the three-layer new feature map into the decoupling head prediction structure, perform feature map integration and prediction data decoding calculation, and obtain prediction bounding box information;

[0212] The posture recognition module is used to preprocess the predicted frame information to obtain the predicted frame and its corresponding confidence, and to perform non-maximum suppression on the predicted frame to obtain the classroom student posture recognition result.

[0213] For the specific functional implementation of each of the above functional modules, please refer to the relevant content in Example 1.

[0214] Example 3

[0215] This embodiment introduces that the present invention provides a classroom student posture recognition device, which includes:

[0216] Memory, for storing computer programs / instructions;

[0217] A processor is used to execute the computer program / instructions to implement the steps of the method for classroom student posture recognition based on dual-trunk feature fusion as introduced in Example 1.

[0218] Example 4

[0219] This embodiment introduces a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for classroom student posture recognition based on dual-trunk feature fusion as introduced in Embodiment 1 are implemented.

[0220] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0222] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0224] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for classroom student posture recognition based on dual-trend feature fusion, characterized in that: include: Get live pictures or videos of students in the classroom; A dual-trunk feature extraction network is used to extract features from live pictures or videos, and each trunk obtains three feature maps of different scales; The feature maps of the same scale obtained from each backbone are concatenated and fused according to the corresponding proportions to obtain three layers of new feature maps; The three layers of new feature maps are input into the decoupling head prediction structure to perform feature map integration and prediction data decoding calculation to obtain the prediction bounding box information; Preprocess the predicted frame information to obtain the predicted frame and its corresponding confidence, and perform non-maximum suppression on the predicted frame to obtain the classroom student posture recognition result; The dual-trunk feature extraction network includes a CNN-based lightweight shuffled backbone network and a Transformer-based sliding window visual backbone network; The CNN-based lightweight shuffle backbone network includes a shuffle space downsampling unit module and a shuffle basic unit module; The Transformer-based sliding window visual backbone network includes a visual segmentation layer, a linear encoding module, a window grouping module A and a window grouping module B; The CNN-based lightweight shuffle backbone network and the Transformer-based sliding window visual backbone network are spliced ​​together through the features extracted by each backbone network; The operation on the sampling unit module in the shuffle space is: S1, the input feature map X1 is first operated by the basic convolution module, the basic convolution module point convolution + batch normalization + ReLu activation function, and then the downsampling depth separable convolution module is operated. The downsampling depth separable convolution module is depth separable convolution + batch normalization operation, and then the basic convolution module is operated again to obtain the output feature map X11; S2, first perform downsampling depth-separable convolution module operation on the input feature map X1, and then perform basic convolution module operation to obtain feature map X12; S3, concatenate X11 and X12 in the channel dimension to obtain the feature map Y1; S4, shuffle the feature map Y1 in the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as ,in, is the feature map obtained by shuffling the feature map Y1 in the channel dimension, group is the number of groups, is the batch size, h and w are the height and width of feature map Y1 respectively, c is the number of channels of feature map Y1, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped; The calculation operation in the shuffle basic unit module is: S1, the feature map obtained from the shuffled space downsampling unit module First divide the branches into , and divide it into , in, For the feature map The first branch obtained by branch division is For the feature map The second branch obtained by branch division is The input feature map The number of channels; S2, the first branch First, perform the basic convolution module operation, and then perform the depthwise separable convolution module operation. The depthwise separable convolution module is a depthwise separable convolution + batch normalization operation, and then perform the basic convolution module operation to obtain the operation output. ; S3, and Concatenate the channel dimensions to get the feature map Y s ; S4, for feature map Y s Perform a shuffle operation on the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as , where group is the number of groups, is the batch size, h and w are the feature map height and width respectively, C s is the feature map Y s The number of channels, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped, and the feature map is finally output , is the feature map sequence output by the shuffled basic unit module, including multiple feature maps [Os1, Os2, Os3] for subsequent feature fusion operations; The visual segmentation layer transforms the input feature map X1 using convolution operation to obtain the output feature map X1 p , where H and W represent the height and width of the feature map X1 respectively; The linear coding module includes a linear coding layer and two sliding window vision modules. The linear coding layer is a feature map X1 obtained after the visual segmentation layer operation. p Follow X1 p’ =X1 p E+E pos Perform a linear projection operation, where E is the learnable weight matrix of the linear projection, E pos It's with X1 p The position-dependent learnable position embedding with D dimensions is transformed into the original X1 through linear projection. p The sequence is mapped to X1 with D dimensions p’ sequence; The window grouping module A includes a window segmentation layer and two sliding window vision modules, and the window grouping module B includes a window segmentation layer and six sliding window vision modules; The window segmentation layer is based on Generate window sequences of different scales in the following way , and then realize the downsampling operation, that is, the feature map height and width are halved and the channel is doubled, where is the convolution kernel of the current operation scale, is the cross-layer convolution kernel of the current operation scale, LN is the layer normalization operation, k is the accumulated variable from 0 to i-1, and n is the total number of layers of window segmentation; The sliding window vision module includes a serially connected window attention module and a shifted window attention module; The operation on the window attention module is: S1, for the input sequence Perform layer normalization to provide data distribution for subsequent operations; S2, the normalized sequence Divide into M windows , the standard multi-head self-attention is calculated in each window, which is calculated as: ; in, is a linear projection, calculating the output attention feature map ; W q , W k , W v is a learnable weight matrix, which is used to generate query vector, key vector and value vector respectively; T represents transposition, represents the scaling factor, and B represents the relative position bias; S3, and Perform residual connection to obtain ’ ; S4, yes ’ Normalize it and get ' ; S5, ' Input feedforward network, according to X 1F =σ( 'W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained 1F ; S6, ' With X 1F Perform residual connection to get X 1F ’ , as the input of the next module; The shift window attention module expands step S2 of the window attention module and introduces a window shift operation based on the window attention module. The specific operation is: S1, for the input sequence X 1F ’ Perform layer normalization to provide data distribution for subsequent operations; S2, the normalized sequence X 1F ’ First perform the shift operation to generate the shifted version X 1F ’ _s; then the shifted version of X 1F ’ _s is processed according to the calculation method of the window attention layer in step S2 of the window attention module to calculate the output attention feature map _s; the last _s performs reverse shift operation and restores to ; S3, and Perform residual connection to obtain ’ ; S4, yes ’ Normalize it and get '; S5, ' Input feedforward network, according to X 1FF =σ( ' W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained. 1FF ; S6, ' With X 1FF Residual connection is performed to obtain the feature map Ow. The final output feature map Ow is a feature map sequence output by the sliding window visual backbone network, including multiple feature maps [Ow1, Ow2, Ow3], which are used for subsequent feature fusion operations.

2. The method according to claim 1, characterized in that The feature maps of the same scale obtained from each backbone are concatenated and fused to obtain three layers of new feature maps. include, Obtain feature maps Ow1, Ow2, and Ow3 of three scales from the sliding window visual backbone network, a total of three feature maps, The feature maps of the same scale are concatenated respectively, i.e. Concat (Os1, Ow1), Concat (Os2, Ow2), Concat (Os3, Ow3), to obtain feature maps O1, O2, O3. The feature maps O1, O2, O3 are passed through the attention multi-layer residual convolution module to obtain feature maps P1, P2, P3; Among them, the specific operations on the attention multi-layer residual convolution module are: After a basic convolution block, the basic convolution block includes a convolution layer + batch normalization layer + SiLU activation function; Use the chunk function to split the output into two vectors evenly; input the second half of the vector into the attention multi-layer residual convolution group, which contains n attention residual convolution units, and append the output of each attention residual convolution unit to the result; After a basic convolution block output, the attention residual convolution unit consists of two basic convolution blocks connected in series with the polarized self-attention mechanism, and residual fusion is performed with the input.

3. The method according to claim 2, characterized in that The three-layer new feature map is input into the decoupling head prediction structure, and feature map integration and prediction data decoding calculation are performed to obtain prediction border information, including: The feature maps P1, P2, and P3 are input into a decoupling head prediction structure, which includes a detection head branch and a classification head branch; both branches first pass through a basic convolution block, and then the detection head obtains a regression branch with a maximum number of channels of 4x16 or the number of channels divided by 4 through a convolution layer; the classification head obtains a classification branch with a number of channels equal to the number of classification categories through a convolution layer; the final output channel number is the sum of the channel numbers of the two branches, and the results of multiple scale feature maps are integrated together to obtain feature prediction data; The feature prediction data is split according to the detection and classification information into classification prediction feature data and position prediction feature data, corresponding to the specifications of batch size × sum of feature map size × number of classification head channels and batch size × sum of feature map size × number of detection head channels respectively; Calculate the scaling factor of the corresponding scale feature map, the scaling factor is the result of dividing the feature map height by the input image height, take each pixel in the feature map as an anchor point, and take the center coordinates of the pixel point as the coordinates of the center of the anchor point; discretize the classification prediction feature data through Softmax, calculate the probability value of the distance of each anchor point center point to the left, top, right, and bottom of the feature map center point, and obtain l, t, r, and b after weighted summation, which respectively represent the distance values ​​of each anchor point center point to the left, top, right, and bottom of the feature map center point; the coordinates of the feature map center point are (x, y) and calculate the upper left corner coordinates of the predicted bounding box in the feature map and the lower right corner coordinates , the specific calculation is: ; ; The position prediction feature data is decoded into batch size × sum of feature map size × 4, and the predicted bounding box information is obtained by combining the processed classification prediction feature data and position prediction feature data.

4. The method according to claim 3, characterized in that The step of preprocessing the predicted frame information to obtain the predicted frame and its corresponding confidence level includes: The position prediction feature data, center coordinates and scaling factor in the prediction frame information are multiplied to convert the position prediction feature data from the normalized scale to the input image scale to obtain the position information; the classification prediction feature data is normalized by the Sigmoid function to obtain the maximum probability category and probability value; the position information and the maximum probability category and probability value are integrated to obtain the prediction frame and the final confidence.

5. The method according to claim 4, characterized in that The prediction box is subjected to non-maximum suppression to obtain the classroom student posture recognition results, including: Arrange the prediction boxes from large to small according to their confidence to form a candidate list; Select the prediction box with the largest confidence and put it into the output list, and delete the selected prediction box from the candidate list; Calculate the intersection-and-union ratio of the current prediction box and the candidate boxes in the candidate list, and delete the candidate boxes whose intersection-and-union ratio is greater than the set threshold from the candidate list; Repeatedly delete the prediction box with the highest confidence and the candidate box with an intersection-over-union ratio greater than the set threshold until the candidate list is completely cleared; Get the final output list; The predicted boxes in the final output list are used as the final classroom student posture recognition results.

6. A classroom student posture recognition system based on dual-trend feature fusion, characterized in that: include: The acquisition module is used to obtain live pictures or videos of students in the classroom; A feature extraction module is used to extract features from live pictures or videos using a dual-trunk feature extraction network, where each trunk obtains three feature maps of different scales; The feature fusion module is used to concatenate and fuse the feature maps of the same scale obtained from each backbone according to the corresponding proportion to obtain three layers of new feature maps; A prediction information acquisition module is used to input the three-layer new feature map into the decoupling head prediction structure, perform feature map integration and prediction data decoding calculation, and obtain prediction bounding box information; The posture recognition module is used to pre-process the predicted frame information to obtain the predicted frame and its corresponding confidence, and to perform non-maximum suppression on the predicted frame to obtain the classroom student posture recognition result; The dual-trunk feature extraction network includes a CNN-based lightweight shuffled backbone network and a Transformer-based sliding window visual backbone network; The CNN-based lightweight shuffle backbone network includes a shuffle space downsampling unit module and a shuffle basic unit module; The Transformer-based sliding window visual backbone network includes a visual segmentation layer, a linear encoding module, a window grouping module A and a window grouping module B; The CNN-based lightweight shuffle backbone network and the Transformer-based sliding window visual backbone network are spliced ​​together through the features extracted by each backbone network; The operation on the sampling unit module in the shuffle space is: S1, the input feature map X1 is first operated by the basic convolution module, the basic convolution module point convolution + batch normalization + ReLu activation function, and then the downsampling depth separable convolution module is operated. The downsampling depth separable convolution module is depth separable convolution + batch normalization operation, and then the basic convolution module is operated again to obtain the output feature map X11; S2, first perform downsampling depth-separable convolution module operation on the input feature map X1, and then perform basic convolution module operation to obtain feature map X12; S3, concatenate X11 and X12 in the channel dimension to obtain the feature map Y1; S4, shuffle the feature map Y1 in the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as ,in, is the feature map obtained by shuffling the feature map Y1 in the channel dimension, group is the number of groups, is the batch size, h and w are the height and width of feature map Y1 respectively, c is the number of channels of feature map Y1, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped; The calculation operation in the shuffle basic unit module is: S1, the feature map obtained from the shuffled space downsampling unit module First divide the branches into , and divide it into , in, For the feature map The first branch obtained by branch division is For the feature map The second branch obtained by branch division is The input feature map The number of channels; S2, the first branch First, perform the basic convolution module operation, and then perform the depthwise separable convolution module operation. The depthwise separable convolution module is a depthwise separable convolution + batch normalization operation, and then perform the basic convolution module operation to obtain the operation output. ; S3, and Concatenate the channel dimensions to get the feature map Y s ; S4, for feature map Y s Perform a shuffle operation on the channel dimension to obtain the feature map , the shuffle operation in the channel dimension is calculated as , where group is the number of groups, is the batch size, h and w are the feature map height and width respectively, C s is the feature map Y s The number of channels, This means that Y1 is The shape of is rearranged and flattened into a 5-dimensional tensor. Transpose means In the results The dimensions and The positions of the dimensions are swapped, and the feature map is finally output , is the feature map sequence output by the shuffled basic unit module, including multiple feature maps [Os1, Os2, Os3] for subsequent feature fusion operations; The visual segmentation layer transforms the input feature map X1 using convolution operation to obtain the output feature map X1 p , where H and W represent the height and width of the feature map X1 respectively; The linear coding module includes a linear coding layer and two sliding window vision modules. The linear coding layer is a feature map X1 obtained after the visual segmentation layer operation. p Follow X1 p’ =X1 p E+E pos Perform a linear projection operation, where E is the learnable weight matrix of the linear projection, E pos It's with X1 p The position-dependent learnable position embedding with D dimensions is transformed into the original X1 through linear projection. p The sequence is mapped to X1 with D dimensions p’ sequence; The window grouping module A includes a window segmentation layer and two sliding window vision modules, and the window grouping module B includes a window segmentation layer and six sliding window vision modules; The window segmentation layer is based on Generate window sequences of different scales in the following way , and then realize the downsampling operation, that is, the feature map height and width are halved and the channel is doubled, where is the convolution kernel of the current operation scale, is the cross-layer convolution kernel of the current operation scale, LN is the layer normalization operation, k is the accumulated variable from 0 to i-1, and n is the total number of layers of window segmentation; The sliding window vision module includes a serially connected window attention module and a shifted window attention module; The operation on the window attention module is: S1, for the input sequence Perform layer normalization to provide data distribution for subsequent operations; S2, the normalized sequence Divide into M windows , the standard multi-head self-attention is calculated in each window, which is calculated as: ; in, is a linear projection, calculating the output attention feature map ; W q , W k , W v is a learnable weight matrix, which is used to generate query vector, key vector and value vector respectively; T represents transposition, represents the scaling factor, and B represents the relative position bias; S3, and Perform residual connection to obtain ’ ; S4, yes ’ Normalize it and get ' ; S5, ' Input feedforward network, according to X 1F =σ( 'W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained 1F ; S6, ' With X 1F Perform residual connection to get X 1F ’ , as the input of the next module; The shift window attention module expands step S2 of the window attention module and introduces a window shift operation based on the window attention module. The specific operation is: S1, for the input sequence X 1F ’ Perform layer normalization to provide data distribution for subsequent operations; S2, the normalized sequence X 1F ’ First perform the shift operation to generate the shifted version X 1F ’ _s; then the shifted version of X 1F ’ _s is processed according to the calculation method of the window attention layer in step S2 of the window attention module to calculate the output attention feature map _s; the last _s performs reverse shift operation and restores to ; S3, and Perform residual connection to obtain ’ ; S4, yes ’ Normalize it and get '; S5, ' Input feedforward network, according to X 1FF =σ( ' W1 + b1)W2 + b2, where W1 and W2 are linear layers, σ is the GELU nonlinear function, and the feedforward feature X is obtained. 1FF ; S6, ' With X 1FF Residual connection is performed to obtain the feature map Ow. The final output feature map Ow is a feature map sequence output by the sliding window visual backbone network, including multiple feature maps [Ow1, Ow2, Ow3], which are used for subsequent feature fusion operations.

7. A classroom student posture recognition device, comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the classroom student posture recognition method based on dual-trunk feature fusion as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for classroom student posture recognition based on dual-trunk feature fusion as described in any one of claims 1 to 5.