Image-based teaching scene behavior recognition network processing method and device

Abstract

The application discloses an image-based teaching scene behavior recognition network processing method and device, and belongs to the technical field of image recognition. The application is used for behavior recognition in a teaching scene. Global and local information is extracted from an image, and the two kinds of information are fused to obtain fused features with a large amount of information. Finally, the features are used to recognize behaviors in the image. The image-based teaching scene behavior recognition method can make up for the problem that a large amount of calculation power is needed in a video stream-based behavior recognition method, and the amount of training data required by the method is smaller. Decoupling of global and local features can obtain more image information than single-channel feature extraction. Through feature fusion and feature concatenation of different layers, more effective information can be obtained from a large amount of extracted initial information. A position recovery module can enhance the position information perception ability of the model for the image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and particularly relates to an image-based teaching scene behavior recognition network processing method and device. BACKGROUND

[0002] Behavior recognition is to recognize the behavior action of a person in an image or a video clip; existing behavior recognition methods are mostly based on short videos, which start from the perspective of time and space analysis, and the specific operation is to extract more useful information from the changes between frames to help behavior recognition, although the recognition accuracy of this kind of method is acceptable, but it requires a large amount of calculation; and the image-based behavior recognition starts from the details of the image, and tries to extract more useful information from the single frame image with limited information, through the capture of detail features and the fusion of global features, so as to obtain the information that is most helpful to behavior recognition.

[0003] Since the amount of information carried by an image is much less than that of a video, pure global feature extraction may not be enough for behavior recognition; therefore, new local features need to be introduced on the basis of the original global features to strengthen the perception ability of the model to the key feature positions, so as to improve the accuracy of behavior recognition. SUMMARY

[0004] The present application provides an image-based teaching scene behavior recognition method, which realizes behavior recognition in a teaching scene, extracts global and local information from an image, and fuses the two kinds of information to obtain fused features with large information quantity; and finally recognizes the behavior in the image according to the features.

[0005] In one aspect, the present application discloses an image-based teaching scene behavior recognition method, which comprises the following steps:

[0006] inputting the collected person image into a global feature extraction network to obtain multi-layer global features;

[0007] using image destruction processing (spatial information destruction processing) to do block division and random rearrangement processing on the collected person image, and then inputting the rearranged image into a local feature extraction network to obtain local features;

[0008] wherein, the image destruction processing is: first dividing the input image into N*N blocks, and then placing the image block with position label P(i,j) at P(l,m); wherein, the position coordinates (i,j) and (l,m) can be the same, but the new positions P(l,m) corresponding to each block cannot overlap;

[0009] the processing of the global feature extraction network is: inputting the original image into a backbone network to obtain multi-layer global feature representations;

[0010] The processing of the local feature extraction network is: inputting the damaged image into the backbone network to obtain a multi-layer local feature representation;

[0011] Feature fusion map acquisition: taking out the features of different layers in the global feature extraction network and the features of different layers in the local feature extraction network to obtain a plurality of fused features (preliminary fusion map) respectively, then obtaining the final feature fusion map through the method of downsampling and cascading convolution; then expanding the obtained features into a feature vector, and then inputting the expanded feature vector into the classification network of the full connection layer to obtain the final classification result; that is, the classification of different behaviors in the teaching scene;

[0012] Position information reconstruction: downsampling the local features of different layers to a unified size and then cascading and combining them into a new feature representation; then obtaining a table containing position reconstruction information through position recovery convolution operation on the obtained new feature representation;

[0013] Supervised coefficient setting: comparing the final classification result with the label to obtain a classification loss, then multiplying the obtained classification loss by the corresponding supervised coefficient to obtain a new classification loss; comparing the position information obtained by position information reconstruction with the template used for spatial information damage processing to obtain a position reconstruction loss; finally, weighting and merging the losses of the two parts into a total loss to train the network parameters of the global feature extraction network, the local feature extraction network, the network used for feature fusion map acquisition, and the network involved in position information reconstruction; when the preset training end condition is met, an identification model for teaching scene behavior recognition is obtained; and then the identification result of the target image is obtained based on the identification model.

[0014] On the other hand, the application also discloses an image-based teaching scene behavior recognition device, which comprises: an image preprocessing module, an image damage module, a feature extraction backbone network, a feature fusion module, a behavior classification module and a training module;

[0015] Among them,

[0016] The image preprocessing module is used for image preprocessing (including size normalization, image format conversion, etc.) of the image to obtain a raw image to be processed, and inputs the raw image into the feature extraction backbone network and the image damage module respectively;

[0017] The image damage module uses a preset damage matrix to perform position damage rearrangement on the input raw image in a block structure, and inputs the rearranged image into the feature extraction backbone network;

[0018] The feature extraction backbone network is used for multi-layer feature extraction of the input image; if the current input is the original image, i.e. the image processed by the image preprocessing module, the output feature representation is a multi-layer global feature and is input into the feature fusion module; if the current input is the image after position destruction rearrangement, the extracted feature represents a multi-layer local feature and is input into the feature fusion module and the training module;

[0019] The feature fusion module adopts a multi-level bilinear fusion structure, which first performs bilinear fusion on the global and local features of different layers, then performs down-sampling on the features after fusion of different layers to a unified size, and then concatenates all the features of the unified size to input into a convolution layer to obtain the final fusion result and input into the behavior classification module. This structure can effectively fuse global and local information, and can fuse global and local information of different levels.

[0020] The behavior classification module expands the input fusion result into a (one-dimensional) feature vector, and then obtains the final classification result (outputs the probability of each behavior category, and the maximum classification probability is the current predicted behavior category) through a fully connected layer classification network. The teaching scene behavior recognition result of the target image is obtained based on the output of the behavior classification module.

[0021] The training module includes a position recovery module and a supervision coefficient setting module.

[0022] The position recovery module is used to down-sample the input multi-layer local features to a unified size, and then concatenate the features of the unified size through convolution to obtain a position recovery matrix. The introduction of the position recovery module can enhance the model's perception of image position information while extracting local features.

[0023] The supervision coefficient setting module compares the classification result output by the behavior classification module with the corresponding label to obtain a classification loss, and then multiplies the obtained classification loss by the corresponding supervision coefficient to obtain a new classification loss. The position recovery matrix obtained by the position recovery module is compared with the destruction matrix to obtain a position reconstruction loss. The position reconstruction loss and the new classification loss are weighted and combined as a total loss, which is used for network parameter training (network parameter optimization, i.e. updating network parameters, such as gradient descent method) of the network parameters of the feature extraction backbone network, the feature fusion module and the behavior classification module. When the preset training end condition (such as the number of training times, the total loss) is met, the training is stopped. The target image is processed by the image preprocessing module, the image destruction module, the trained feature extraction backbone network, the feature fusion module and the behavior classification module to obtain the teaching scene behavior recognition result of the target image.

[0024] In summary, due to the adoption of the above technical solutions, the present application has the following advantages:

[0025] 1. The image-based teaching scene behavior recognition method can make up for the large computing power requirement of the video stream-based behavior recognition method, and the training data required by the method is also smaller.

[0026] 2. Decoupling global and local features for separate extraction can obtain more image information than single-path feature extraction.

[0027] 2. By fusing and cascading features of different layers, more effective information can be obtained from a large amount of extracted initial information.

[0028] 3. The position recovery module can enhance the model's ability to perceive the position information of the image. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 In the embodiments of the present application, the behavior recognition model structure diagram is adopted. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0032] As shown in Figure 1 The behavior recognition model for realizing teaching scene behavior recognition of the present application includes: feature extraction backbone, feature fusion module, behavior classification module, position recovery module, image destruction and reconstruction module.

[0033] Feature extraction backbone: CNN structure is adopted to extract global and local features from the original image and the destroyed image respectively. By separately extracting two kinds of features, more effective information can be obtained from the input image.

[0034] Feature fusion module: a multi-level bilinear fusion structure is adopted, which first performs bilinear fusion on the global and local features of different layers, then down-samples the features after fusion of different layers to a unified size, and then cascades all the features of the same size to input into the convolution layer to obtain the final fusion result. This structure can effectively fuse global and local information, and can fuse global and local information of different levels.

[0035] The behavior classification module expands the fusion result input by the feature fusion module into a (one-dimensional) feature vector, and obtains the final classification result (outputs the probability of each behavior category, and the maximum classification probability is the predicted behavior category) through a fully connected layer classification network.

[0036] The position recovery module: first, the input multi-layer local features are down-sampled to a uniform size, then the features with uniform size are concatenated, and then a position recovery matrix is obtained through convolution. The introduction of the position recovery module can enhance the perception ability of the model to image position information while the supervised model extracts local features.

[0037] The image destruction module: using a previously generated destruction matrix to do block structure position destruction rearrangement on the input original image; the image reconstruction module: using the destruction matrix to recover the original image from the destroyed image.

[0038] Next, the parts of the network will be described in detail:

[0039] (1) Global and local feature extraction

[0040] The present application is based on an assumption that the key discriminant area of behavior recognition only accounts for a small part of the feature area of the whole image. The original image is cut and position destroyed to remove as much global structured information of the image as possible, and the ability of the feature extraction network to extract local information in the image is enhanced in the subsequent local feature extraction process.

[0041] The image destruction is represented as follows:

[0042] I' = D(I, M g )

[0043] The image reconstruction is represented as follows:

[0044] I = R(I', M g )

[0045] The local feature extraction is represented as follows:

[0046]

[0047] The global feature extraction is represented as follows:

[0048]

[0049] Where D is the image destruction operation, R is the image reconstruction operation, f i is the feature extractor, i is the number of network layers, M g is the destruction matrix, and I is the original image.

[0050] (2) Feature fusion and classification.

[0051] The application is based on a bilinear feature fusion technology. The bilinear feature fusion method can effectively fuse two different features through outer product. The specific implementation scheme is as follows: the local and global features of different layers are respectively subjected to feature fusion, then the features of different sizes obtained after fusion are respectively subjected to corresponding down-sampling to unify the sizes, finally all the features are cascaded and then subjected to convolution to obtain the fused result.

[0052] The feature fusion is expressed as follows:

[0053]

[0054]

[0055] The supervision intensity coefficient is expressed as follows:

[0056]

[0057] The classification loss is expressed as follows:

[0058]

[0059] wherein Fusion represents a single-layer feature fusion operation, represents the global feature of the i-th layer, represents the local feature of the i-th layer, g represents a feature fusion function, represents the fused i-th layer feature, K represents the number of layers, β represents a coefficient (empirical value) for controlling the supervision intensity, C represents the total number of classes of a data set, is the proportion of the sample with the class number j, represents the label (the number of the behavior class to which the sample belongs) of the sample with the number j, represents the predicted class of the network output, i.e. the class output by the behavior classification module.

[0060] (3) Position recovery.

[0061] Since the cut and reorganization operation is adopted on the original image, such operation can destroy the global structure information of the image, but also destroys the absolute position information of the image, therefore, the position recovery constraint needs to be applied to the features extracted from the destroyed image, so as to enhance the learning ability of the model to the position information.

[0062] The position recovery matrix calculation is expressed as follows:

[0063]

[0064] The position recovery loss is expressed as follows:

[0065]

[0066] wherein, represents the local feature of the i-th layer, h is a position recovery matrix generating function, l is the coordinate axis number of the matrix, and N is the number of patches.

[0067] In the training stage, the global information of the input image is first destroyed using the image destruction module, then the two images are respectively input into the network to obtain global feature and local feature representation, and then the feature fusion module is used to fuse the global and local features of each layer, and finally the fused features are input into the behavior classification network to classify the input image. The position recovery matrix is recovered from the extracted local feature by using the position recovery module, and the position recovery matrix and the input position destruction matrix are subjected to two norm loss constraint. In addition, the M r and M g Taking normalization processing can reduce the learning difficulty of the model. In the inference stage, the position recovery module is removed, and only the feature extraction part, the feature fusion module and the classification part of the model are retained.

[0068] In this embodiment, the recognized behaviors are 13 categories, including 10 student behaviors (listening to class, writing, reading, standing, etc.) and 3 teacher behaviors (writing on the board, teaching, and displaying).

[0069] In summary, the scheme proposed in the present application can combine the global and local features of the image; the model size is basically consistent with the general method, and the two features are decoupled by using the two features respectively extracted and then fused, which enhances the perception ability of the model to the small changes in the image. The multi-level feature fusion method used in the present application can enhance the multi-scale perception ability of the model to the features. The introduction of the position recovery module can improve the perception ability of the model to the feature position. The supervision intensity coefficient can adjust the supervision intensity of different categories of samples, and improve the robustness of the model to the sample distribution.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0071] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A network processing method for image-based teaching scene behavior recognition, characterized in that, Includes the following steps: The collected images of people are input into a feature extraction network to obtain multi-layer global features; The collected images of people are processed by image destruction to perform block segmentation and random rearrangement. The rearranged images are then input into a feature extraction network to obtain multi-layer local features. The image destruction process involves dividing the input image into N×N blocks and then repositioning the image block labeled P(i,j) at position P(l,m). The position coordinates (i,j) and (l,m) can be the same, but the new positions P(l,m) corresponding to each block cannot overlap. Feature fusion map acquisition: Global and local features from different layers are extracted and fused to obtain multiple preliminary fused features. The preliminary fused features are then downsampled and concatenated convolutions to obtain the final feature fusion map. The feature fusion map is then expanded into feature vectors, and the expanded feature vectors are input into a fully connected classification network to obtain the behavior classification results of the teaching scenario. Location information reconstruction: Local features from different layers are downsampled to a uniform size and then concatenated to form a new feature representation; the new feature representation is then subjected to a location recovery convolution operation to obtain the location recovery matrix; Supervision coefficient settings: The classification loss is obtained by comparing the behavior classification result output by the classification network with the label. The new classification loss is then multiplied by the corresponding supervision strength coefficient. The location recovery matrix obtained by reconstructing the location information is compared with the location destruction information used in image destruction processing to obtain the location reconstruction loss. The new classification loss and the location reconstruction loss are weighted and merged into the total loss. The network parameters of the global feature extraction network, the local feature extraction network, the network used for feature fusion map acquisition, and the network involved in location information reconstruction are trained. When the preset training termination condition is met, a recognition model for behavior recognition in teaching scenarios is obtained. The recognition result of the target image is obtained based on this recognition model.

2. The method as described in claim 1, characterized in that, The new classification loss is: Where, α j This represents the supervision strength coefficient of the sample with category number j. This represents the behavioral classification label for sample j. This represents the predicted class output by the classification network, where C is the total number of classes in the dataset.

3. The method as described in claim 2, characterized in that, The supervision intensity coefficient is set as follows: Where β represents the coefficient for controlling the intensity of supervision. This indicates the proportion of samples with category number j.

4. An image-based teaching scene behavior recognition device, characterized in that, include: The system includes an image preprocessing module, an image destruction module, a feature extraction backbone network, a feature fusion module, an behavior classification module, and a training module. in, The image preprocessing module is used to preprocess the image to obtain the original image to be processed, and input the original image into the feature extraction backbone network and the image destruction module respectively. The image destruction module uses a preset destruction matrix to perform position destruction and rearrangement of the block structure of the input original image, and then inputs the rearranged image into the feature extraction backbone network. The feature extraction backbone network is used to extract multi-layer features from the input image. If the current input is the original image, the output feature representation is multi-layer global features and is input into the feature fusion module. If the current input is an image with positional disruption and rearrangement, the extracted feature representation is multi-layer local features and is input into the feature fusion module and the training module. The feature fusion module adopts a multi-level bilinear fusion structure. This structure first performs bilinear fusion on global and local features at different layers, then downsamples the fused features at different layers to a uniform size, and then concatenates all features of uniform size and inputs them into a convolutional layer to obtain the final fusion result and input it into the behavior classification module. This structure can effectively fuse global and local information and can fuse global and local information at different levels. The behavior classification module expands the input fusion result into a feature vector, and then passes the expanded feature vector through a fully connected classification network to obtain the behavior classification result for the teaching scenario. The training module includes a location recovery module and a supervision coefficient setting module; The position recovery module is used to downsample the input multi-layer local features to a uniform size, and then concatenate the features of uniform size through convolution to obtain the position recovery matrix. The supervision coefficient setting module compares the classification result output by the behavior classification module with the corresponding label to obtain the classification loss, and then multiplies the obtained classification loss by the corresponding supervision coefficient to obtain the new classification loss; it compares the position recovery matrix obtained by the position recovery module with the destruction matrix to obtain the position reconstruction loss; it weights and combines the position reconstruction loss with the new classification loss to obtain the total loss, which is used to fine-tune the network parameters of the feature extraction backbone network, feature fusion module, and behavior classification module. It stops when the preset parameter tuning termination condition is met. The target image is processed through an image preprocessing module, an image destruction module, a trained feature extraction backbone network, a feature fusion module, and a behavior classification module to obtain the teaching scene behavior recognition result of the target image; The destruction matrix is ​​obtained by dividing the input image into N×N blocks, and then repositioning the image block with position number P(i,j) to P(l,m). The position coordinates (i,j) and (l,m) can be the same, but the new positions P(l,m) corresponding to each block cannot overlap. The destruction matrix is ​​determined based on the position of the repositioned image block.

Images