Work behavior recognition method and device, electronic equipment and storage medium

Abstract

The application provides a kind of working behavior recognition method, device, electronic equipment and storage medium, belong to image recognition technical field, including: obtaining the working behavior image sequence to be identified;Working behavior image sequence is input into the working behavior recognition model trained, obtains the working behavior classification result that working behavior recognition model outputs;Wherein, working behavior recognition model is the series connection hybrid model based on convolutional neural network and Transformer model.The application uses the series connection hybrid model based on convolutional neural network and Transformer model to identify working behavior image, and convolutional neural network can extract local features from working behavior image, and the Transformer model can establish the relationship between the global features of working behavior image, so that the working behavior recognition model has high accuracy in identifying working behavior, and accurate working behavior classification results can be obtained, thereby improving the accuracy of working behavior recognition based on existing technology.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, electronic device, and storage medium for recognizing labor behavior. Background Technology

[0002] In the field of agricultural production, many labor activities are still carried out manually by the workers, such as watermelon planting in the field. Sometimes it is necessary to supervise the workers' labor activities to determine whether the workers have carried out the labor activities and whether the labor activities are correct. The supervision of labor activities requires accurate identification of the labor activities.

[0003] Labor behavior identification methods include manual visual identification and image recognition. Image recognition is widely used due to its advantages of being independent of manual labor, highly automated, and efficient. However, current labor behavior image recognition methods are limited by the accuracy of the models, resulting in low accuracy in labor behavior identification. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for recognizing labor behavior, which solves the technical problem of low accuracy in labor behavior recognition in the prior art and improves the accuracy of labor behavior recognition; and uses a shallower model layer to reduce the number of parameters.

[0005] This invention provides a method for recognizing work behavior, comprising: Obtain the image sequence of the labor behavior to be identified; The sequence of images of labor behavior is input into a trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model. The labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

[0006] According to a method for recognizing work behavior provided by the present invention, the step of acquiring a sequence of work behavior images to be recognized includes: Acquire video data of labor activities captured by drones; Extract the image sequence of the labor activity from the video data of the labor activity.

[0007] According to the present invention, a method for recognizing labor behavior is provided, wherein the labor behavior recognition model includes an input layer, a convolutional layer, a reshaping layer, a Transformer Encoder layer, a global average pooling layer, and a classification layer; The input layer is used to input the sequence of labor behavior images; the convolutional layer uses a convolutional neural network to extract feature maps from the sequence of labor behavior images; the reshaping layer stacks the feature maps into a feature sequence; the Transformer Encoder layer processes the feature sequence; the global average pooling layer performs global average pooling on the output of the Transformer Encoder layer; and the classification layer generates a classification result of the labor behavior based on the output of the global average pooling layer.

[0008] According to a method for identifying labor behaviors provided by the present invention, the classification results of the labor behaviors include turning over soil, sowing seeds, weeding, spraying pesticides, or emasculating.

[0009] According to a method for recognizing work behavior provided by the present invention, before inputting the work behavior image sequence into a trained work behavior recognition model to obtain the work behavior classification result output by the work behavior recognition model, the method further includes: Establish the initial model of the labor behavior recognition model, the training set of labor behavior images, and the test set of labor behavior images; The training set of the labor behavior images is input into the initial model, and the initial model is trained using the Adam optimizer and the classification cross-entropy loss function. The labor behavior image test set is input into the trained initial model for testing, and the initial model that passes the test is used as the labor behavior recognition model.

[0010] According to a method for recognizing work behavior provided by the present invention, the steps of establishing a training set and a test set of work behavior images include: Obtain image samples of labor activities; The image samples of the labor behavior are data augmented to obtain augmented image samples; the data augmentation includes one or more of the following: brightness adjustment, rotation, horizontal flipping, and vertical flipping. A dataset consisting of the labor behavior image samples and the enhanced image samples is established; The dataset is divided into a training set of labor behavior images and a test set of labor behavior images.

[0011] The present invention also provides a labor behavior recognition device, comprising: The acquisition module is used to acquire image sequences of the labor behavior to be identified; The recognition module is used to input the sequence of labor behavior images into a trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model. The labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the labor behavior recognition method as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the labor behavior recognition method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the labor behavior recognition method as described above.

[0015] The present invention provides a method, apparatus, electronic device, and storage medium for recognizing labor behavior. It uses a cascaded hybrid model based on convolutional neural networks (CNNs) and Transformer models to recognize labor behavior images. The CNNs can extract local features from the images, capturing details and local patterns. The Transformer model can establish relationships between global features of the images, improving the understanding of complex image patterns. The combination of CNNs and Transformer models results in high accuracy in recognizing labor behavior, leading to accurate classification results and improving the accuracy of labor behavior recognition compared to existing technologies. Furthermore, the labor behavior recognition model consists only of CNNs and Transformer models, using a shallower model layer. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts of the labor behavior recognition method provided by the present invention.

[0018] Figure 2 This is the second flowchart of the labor behavior recognition method provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the labor behavior recognition model provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the structure of the convolutional neural network provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the Transformer Encoder layer provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the structure of the MLP provided by the present invention.

[0023] Figure 7 This is a schematic diagram of the labor behavior image sample provided by the present invention.

[0024] Figure 8 This is a schematic diagram of the labor behavior image sample and the enhanced image sample provided by the present invention.

[0025] Figure 9 This is a schematic diagram comparing the effects of the prior art 3 provided by this invention with the labor behavior recognition model.

[0026] Figure 10 This is a schematic diagram of the confusion matrix of the labor behavior recognition model provided by the present invention.

[0027] Figure 11 This is a schematic diagram of the labor behavior recognition device provided by the present invention.

[0028] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0031] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0032] This section provides a brief overview of existing technologies related to human behavior image recognition. Some existing solutions use convolutional neural networks to identify the type of cyclists captured by drones, achieving an accuracy rate of up to 93.61%. Other solutions use drones to capture images of the human body and propose deep learning models to identify human poses, achieving an accuracy rate of 95.2%. Still others use drones to capture images of the human body and represent human contours with elliptical clusters to identify human movements, achieving action recognition accuracies of 95%, 90%, and 44%, respectively. Finally, some existing solutions employ a sparse weighted temporal attention mechanism for activity recognition based on drone cameras, achieving an accuracy rate of 92.56%.

[0033] This section also provides a brief overview of existing technologies related to image recognition of labor activities.

[0034] Some existing technical solutions employ a model containing eight convolutional layers, five pooling layers, two fully connected layers, and one softmax output layer. All convolutional kernels used in this model are 3×3×3, and all pooling layers employ max pooling. The 3D convolutional kernel size is k×k×d. In extracting work behavior feature information, k×k can extract action space information, and depth d performs convolution operations on the pixels at corresponding positions in the input image sequence and the corresponding feature points in the convolutional layers to establish contextual relationships. (Hereinafter referred to as Existing Technology 1).

[0035] Some existing technical solutions employ a pseudo-3D residual network, utilizing a 1×3×3 two-dimensional spatial convolution kernel and a 3×1×1 one-dimensional temporal convolution kernel to simulate a commonly used 3×3×3 three-dimensional convolution kernel. Compared to Existing Technology 1, this existing technical solution reduces the computational cost and complexity of pure 3D convolution by decomposing the 3D convolution into 2D spatial convolution and 1D temporal convolution, effectively solving a series of problems arising from the increasing network depth in Existing Technology 1. (Hereinafter referred to as Existing Technology 2) Some existing technical solutions have achieved an accuracy rate of 94.4% in identifying farmers' work behavior from a ground perspective, demonstrating good performance in terms of efficiency, generalization ability, and stability. (Hereinafter referred to as Existing Technology 3) Some existing technical solutions improve upon the method of Existing Technology 3 by proposing a lightweight network model based on (2+1)D convolution and residual structures, using Swish as the activation function, thereby increasing the accuracy to 97.5% and reducing the model complexity. (Hereinafter referred to as Existing Technology 4) The following is combined Figures 1-12 The present invention describes a method, apparatus, electronic device, and storage medium for recognizing work behaviors, with the aim of improving the accuracy of work behavior recognition.

[0036] Figure 1 This is a flowchart illustrating the labor behavior recognition method provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps: Step S1: Obtain the image sequence of the labor behavior to be identified; Step S2: Input the sequence of labor behavior images into the trained labor behavior recognition model to obtain the labor behavior classification results output by the labor behavior recognition model; Among them, the labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

[0037] The Transformer model is a deep learning model based entirely on self-attention mechanisms, achieving sequence-to-sequence transformation through an encoder-decoder structure. Convolutional neural networks (CNNs) excel at handling complex image features, particularly local features, but are weaker at processing global information. Transformers, on the other hand, are capable of modeling and generating sequential data, possessing advantages in handling global information, but are relatively weaker at processing local information. Combining CNNs with Transformers effectively captures and processes both local and global information in images, thereby improving the model's performance and effectiveness.

[0038] Understandably, a trained labor behavior recognition model needs to be obtained before step S2, therefore, as Figure 2 As shown, before step S2, the labor behavior recognition method of the present invention may further include: Step S3: Establish the initial model of the labor behavior recognition model, the training set of labor behavior images, and the test set of labor behavior images; Step S4: Input the training set of labor behavior images into the initial model, and train the initial model using the Adam optimizer and the classification cross-entropy loss function; Step S5: Input the labor behavior image test set into the trained initial model for testing, and use the initial model that passes the test as the labor behavior recognition model.

[0039] In step S3, the initial model structure is as follows: Figure 3 As shown, the system includes an input layer, convolutional layers, a reshaping layer, a Transformer Encoder layer, a global average pooling layer, and a classification layer. The input layer is used to input the image. The convolutional layers employ Convolutional Neural Networks (CNNs) to extract feature maps from the input image. The reshaping layer stacks the feature maps into a feature sequence. The Transformer Encoder layer processes the feature sequence. The global average pooling layer performs global average pooling on the output of the Transformer Encoder layer. The classification layer generates a classification result for the work behavior based on the output of the global average pooling layer.

[0040] The convolutional layer structure of the initial model is as follows: Figure 4As shown, the convolutional neural network consists of, from bottom to top, a first max-pooling layer (MaxPooling), a first convolutional layer (Conv), a second max-pooling layer (MaxPooling), and a second convolutional layer (Conv). The image is the input image. The first convolutional layer uses 64 kernels (filters) of size 3x3. Each convolution operation performs a dot product operation between the pixel values ​​of a 3x3 region and the kernel. The activation function is ReLU, and the introduction of non-linearity helps the model better represent complex features. Max-pooling reduces the spatial dimensionality of the feature map, thereby reducing computation and selecting important features to suppress overfitting. Before the second convolution, a residual branch is generated, such as... Figure 4 As shown, the residual branch does not change the spatial dimension of the feature map, but it adjusts the number of channels. The second convolution is similar to the first convolution, but the number of output channels is modified to 128 to capture more features. The output after the second convolution is added to the output of the residual branch. The residual connection can alleviate the gradient vanishing problem in deep neural networks, promote better information propagation, and make deep models easier to train.

[0041] In some implementations, the remodeling layer can use a Stack function to stack the feature maps output by the initial model convolutional layers to form a sequence of feature maps, which can then be used as input to the Transformer Encoder layer.

[0042] The Transformer Encoder layer is used to serialize the feature maps extracted by the convolutional layers and capture the global relationships between different feature maps. Its structure is as follows: Figure 5 As shown, the Transformer Encoder layer, from bottom to top, includes a first normalization layer (Norm), a multi-head attention mechanism layer (Multi-Head Attention), a second normalization layer (Norm), and a feedforward neural network layer (such as a multilayer perceptron, MLP). Embedded Patches are small image patches obtained after segmenting the feature map. The normalization layer (Norm) is used to normalize the feature map sequence. The multi-head attention mechanism layer is the core component of the Transformer Encoder layer, allowing the model to perform multi-view self-attention calculations on the input feature map sequence in parallel on different "heads," enabling the model to simultaneously capture global relationships at different locations in the feature map sequence.

[0043] The structure of MLP is as follows Figure 6As shown, from bottom to top, the model consists of a first linear layer (Linear), an activation layer (ReLU), a regularization layer (Dropout), a second linear layer (Linear), and a second regularization layer (Dropout). The first linear layer (Linear) sets the output dimension to 256 and performs a linear transformation on the input feature map sequence. The activation layer (ReLU) uses the ReLU activation function to perform non-linear activation on the input feature map sequence. The regularization layer (Dropout) normalizes each feature dimension of each feature map, enhancing the model's stability and convergence speed. The second linear layer (Linear) reduces the output dimension to 128. By gradually reducing the feature dimension, the model can more effectively learn and represent the abstract features of the input data, while avoiding overfitting and improving the model's generalization ability.

[0044] The Transformer Encoder layer uses layer normalization to improve the stability and convergence speed of the model when processing feature map sequences, replacing the traditional self-attention mechanism.

[0045] The global average pooling layer is used to perform global average pooling on the feature map sequence. The classification layer is used to output the classification probability distribution, that is, the classification result of the labor behavior. In some implementations, the classification layer can use a multilayer perceptron head (MLPhead) for classification.

[0046] Generally, CNNs perform well in image processing, handling complex image features and excelling at processing local features, but are weaker at processing global information. Transformers, on the other hand, can model and generate sequential data, excelling at processing global information, but are relatively weaker at processing local information. This invention combines CNNs and Transformers to effectively capture and process both local and global information in images, thereby improving the model's performance and effectiveness.

[0047] In some implementations, step S3, which involves establishing a training set and a test set of labor behavior images, may include: Obtain image samples of labor activities; Data augmentation is performed on images of labor activities to obtain augmented image samples; data augmentation includes one or more of the following: brightness adjustment, rotation, horizontal flipping, and vertical flipping. A dataset consisting of labor behavior image samples and augmented image samples was established; The dataset is divided into a training set of images of labor behavior and a test set of images of labor behavior.

[0048] This invention uses watermelon cultivation as an example to illustrate the labor involved in watermelon cultivation. These labor activities include ripping, sowing, weeding, spraying, and emasculating, covering the entire process of watermelon cultivation. Ripping specifically involves the worker stepping on the edge of a shovel and using the shovel head to dig and turn the topsoil downwards, repeating this process. Weeding involves the worker bending down and using weeding tools to remove weeds from the soil. Spraying involves the worker carrying spraying tools and spraying pesticides on the ground or plants. Emasculating involves the worker squatting down and removing the stamens from the flowers.

[0049] In some implementations, the process of obtaining labor behavior image samples is as follows: Classify the various labor behavior videos; process each type of labor behavior video using the open-source computer vision library (OpenCV) built into Python, extracting one image every 5 frames as a keyframe; use the YOLOv8 algorithm to detect the human body region in the keyframes, standardizing the image size to 224×224, and save the human body region image; inspect the extracted human body region images, removing images that do not meet the requirements; in a continuous keyframe group, save every 10 consecutive keyframes as a set of labor behavior image samples, such as... Figure 7 As shown, Figure 7 The first row contains a set of images of labor activities, the second row contains a set of images of labor activities such as tilling, the third row contains a set of images of labor activities such as weeding, and the fourth row contains a set of images of labor activities such as spraying pesticides.

[0050] Since deep learning requires a large amount of data for training, data augmentation is needed on the images of labor behavior to expand the scale and diversity of the dataset, helping the model to better understand and recognize the differences in actions and the interference of other factors.

[0051] The mathematical expression for brightness adjustment is: = x represents an image sample of the labor activity. To enhance image samples, This is the brightness gain coefficient, used to control the proportion of brightness. >1 will brighten the image. A value less than 1 will darken the image. This is a brightness offset used to increase or decrease the overall brightness of an image. By modifying the brightness level of an image by increasing or decreasing the intensity of individual pixels, it helps simulate different lighting conditions that objects or scenes may encounter, thereby improving the model's robustness to changes in lighting.

[0052] The mathematical expression for the rotated image is: = , Let be a rotation matrix. This refers to the rotation angle. Different rotation angles can be selected. It can generate multiple rotated images, enriching the diversity of training data and helping to enhance the model's robustness to images from different angles, enabling it to better identify and classify objects or features appearing in different directions.

[0053] Horizontal image flipping involves mirroring the image along the vertical axis, and its mathematical expression is: = , This is a horizontally flipped matrix.

[0054] Vertical image flipping involves mirroring the image along the horizontal axis, and its mathematical expression is: = , This is a vertically flipped matrix.

[0055] Flipping an image horizontally or vertically can increase the diversity of training data, help the model learn more spatial variation features, improve its ability to recognize objects or scenes in different directions, and enhance the model's generalization ability.

[0056] like Figure 8 The image shown is a sample of a labor activity image and various augmented image samples. Originalimage is a sample of a labor activity image. Brighter makes the image brighter, Darker makes the image darker, Rotation rotates the image, Horizontal flip flips the image horizontally, and Vertical flip flips the image vertically.

[0057] The final sequence of labor behavior image samples (each sequence includes 10 labor behavior image samples), the dataset (labor behavior image samples + augmented image samples), the number of labor behavior image training sets and labor behavior image test sets are shown in Table 1. The dataset contains 20,100 images, of which the training set accounts for 80% (16,080 images) and the test set accounts for 20% (4,020 images).

[0058] Table 1

[0059] The initial model of this invention has an image input shape of (10, 224, 224, 3), representing 10 color images of size 224x224, each with 3 channels. The output shape of the initial model during the experiment is shown in Table 2.

[0060] Table 2

[0061] This invention uses the deep learning model framework Tensorflow 2.4.1 and the programming language Python 3.6 when training the initial model, and CUDA version 11.3 is used for GPU acceleration. The hardware configuration is shown in Table 3.

[0062] Table 3

[0063] To improve the model's generalization ability and the completeness of the experiments during training, K-fold cross-validation was employed. Specifically, the training set of labor behavior images was divided into K subsets. In each training iteration, one subset was selected as the validation set, while the remaining K-1 subsets were used as the training set. The average of the K validation results yielded a more reliable model performance evaluation. Using K-fold cross-validation to evaluate model performance significantly reduces the randomness caused by a single data partition, resulting in a more reliable performance evaluation and greatly reducing the risk of overfitting, thus improving the model's generalization ability.

[0064] This invention uses Adam (Adaptive Moment Estimation) as the optimizer for training. Adam dynamically adjusts the learning rate based on the gradient first-moment and second-moment estimates of each parameter. The initial learning rate is set to 0.001, and training is conducted for 50 epochs (one epoch represents one complete traversal of the entire training set of labor behavior images for the initial model). During model training, categorical cross-entropy is used as the loss function, which effectively measures the difference between the model's predicted class probability distribution and the actual class distribution, prompting the model to learn the correct class distribution.

[0065] This invention calculates the accuracy, mAP, recall, F1 score, and number of parameters used in the statistical model of the labor behavior classification results obtained by the labor behavior recognition model during the training process, and compares them with the prior art 1-4.

[0066] Where, mAP= i is the index of AP, j is the number of APs, and AP = TP represents the number of true positives, indicating that the model correctly classifies positive samples as positive samples; FP represents the number of false positives, indicating that the model incorrectly classifies negative samples as positive samples; therefore, AP represents the proportion of true positives among samples classified as positive, and can comprehensively estimate the model's accuracy and loss value. mAP can measure the model's performance when detecting multiple classes.

[0067] Recall= FN represents the number of positive samples that the model incorrectly classifies as negative samples, while Recall represents the percentage of samples that are actually positive but are correctly predicted as positive by the model. Recall better reflects the model's ability to capture positive examples.

[0068] F1= The F1 score is the harmonic mean of AP and Recall, which is a comprehensive evaluation index that can comprehensively consider the accuracy and completeness of the model.

[0069] The comparison results of existing technologies 1-4 and the labor behavior recognition model (referred to as the CT-WCLNet model) are shown in Table 4.

[0070] Table 4

[0071] It can be seen that the labor behavior recognition model of this invention has significantly improved in all performance indicators and greatly reduced the number of parameters. Compared with the prior art 3, the accuracy has increased by 1.63%, and the training parameters have been reduced by 62.58%. A visual comparison of the accuracy and loss values ​​of the prior art 3 and the CT-WCLNet model during training is shown below. Figure 9 As shown, EPCI represents existing technology 3. It can be observed that after Epochs=10, the CT-WCLNet model tends to converge. Although both curves reach their maximum values ​​during training, the accuracy and loss value of the CT-WCLNet model remain stable in the later stages of training, while existing technology 3 shows obvious fluctuations.

[0072] Furthermore, a confusion matrix for the labor behavior recognition model is established, such as... Figure 10As shown, Predict represents the predicted result, and Actual represents the actual label. A confusion matrix, also known as a probability matrix or error matrix, is used in supervised learning-based classification tasks to compare the differences between the model's predicted results and the actual labels. A normalized confusion matrix is ​​a variant of the confusion matrix, achieved by normalizing each row so that its sum is 1. In supervised learning-based classification tasks, the advantage of a normalized confusion matrix is ​​that it provides a more intuitive and accurate evaluation of model performance. Each row represents the true class, each column represents the predicted class, and the values ​​on the diagonal represent the proportion of correctly classified categories, while the values ​​off-diagonal represent the proportion of incorrectly classified categories. The results displayed by the normalized confusion matrix reveal more deeply the model's performance across different classes, helping to take optimization measures for specific problem categories and thus improve the model's overall classification ability.

[0073] from Figure 10 It can be seen that the correct classification rate of all categories in the labor behavior recognition model is over 90%, and the recognition accuracy of sowing and spraying is as high as 95%.

[0074] As can be seen from the above, this invention uses a cascaded hybrid model based on convolutional neural networks and Transformer models to recognize images of labor behavior. The convolutional neural network can extract local features from the images of labor behavior, capturing details and local patterns; the Transformer model can establish relationships between global features of the images of labor behavior, improving the ability to understand complex image patterns. The combination of convolutional neural networks and Transformer models results in high accuracy in recognizing labor behavior, leading to accurate classification results and improving the accuracy of labor behavior recognition compared to existing technologies. Furthermore, the labor behavior recognition model consists only of convolutional neural networks and Transformer models, using a shallower number of model layers. Experimental results also show that this invention significantly reduces the number of parameters.

[0075] Furthermore, for acquiring image sequences of labor activities, the mainstream image acquisition method is to install fixed cameras in the field. That is, in some implementations, step S1 may include: acquiring video data of labor activities captured by fixed cameras; and extracting image sequences of labor activities from the video data of labor activities captured by fixed cameras.

[0076] However, using cameras to acquire images has significant shortcomings and limitations in practical applications. First, the complex terrain and widespread farmland in mountainous areas necessitate the installation of numerous power poles and lines to provide comprehensive coverage across large melon fields, resulting in significant construction difficulties and substantial costs. Second, fixed cameras, due to their static installation, often have limited monitoring ranges, making it difficult to achieve comprehensive coverage of large areas of farmland. Furthermore, once the installation location of fixed cameras is determined, the viewing angle cannot be flexibly adjusted, making it difficult to adapt to dynamic operational needs within the melon fields, such as changes in worker positions or shifts in focus areas. In addition, the maintenance costs of cameras and their performance stability under adverse weather conditions further complicate and render traditional solutions impractical. Therefore, these issues render traditional camera solutions inadequate for agricultural monitoring in mountainous areas, failing to meet practical needs.

[0077] As a mobile monitoring device, drones offer advantages such as flexible deployment, adjustable viewing angles, and wide coverage, enabling efficient collection of video data on work activities and effectively overcoming many limitations of traditional cameras. Especially in complex terrain and harsh environments, the high mobility and low cost of drones make them even more suitable. Therefore, using drones to capture video data of work activities is a better choice.

[0078] In some implementations, step S1 may include: acquiring video data of labor behavior captured by a drone; and extracting a sequence of labor behavior images from the video data of labor behavior captured by the drone.

[0079] The shooting parameters for using drones to capture video data of labor activities in this invention are shown in Table 5.

[0080] Table 5

[0081] Furthermore, as can be seen from the above, the labor behavior recognition model in step S2 may include an input layer, a convolutional layer, a reshaping layer, a Transformer Encoder layer, a global average pooling layer, and a classification layer. The input layer is used to input the sequence of labor behavior images; the convolutional layer uses a convolutional neural network to extract feature maps from the sequence of labor behavior images; the reshaping layer is used to stack the feature maps into a feature sequence; the TransformerEncoder layer is used to process the feature sequence; the global average pooling layer is used to perform global average pooling on the output of the Transformer Encoder layer; and the classification layer is used to generate the labor behavior classification result based on the output of the global average pooling layer.

[0082] Furthermore, as can be seen from the above, the classification results of labor behaviors in step S2 may include turning the soil, sowing seeds, weeding, spraying pesticides, or emasculating.

[0083] like Figure 11 As shown, the present invention also provides a labor behavior recognition device, comprising: The acquisition module is used to acquire image sequences of the labor behavior to be identified; The recognition module is used to input the sequence of images of labor behavior into the trained labor behavior recognition model and obtain the labor behavior classification results output by the labor behavior recognition model. Among them, the labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

[0084] In some implementations, the acquisition module can also be used for: Acquire video data of labor activities captured by drones; Extract image sequences of labor activities from video data of labor activities.

[0085] In some implementations, the labor behavior recognition model may include an input layer, a convolutional layer, a reshaping layer, a Transformer Encoder layer, a global average pooling layer, and a classification layer; The input layer is used to input the sequence of labor behavior images; the convolutional layer uses a convolutional neural network to extract feature maps from the sequence of labor behavior images; the reshaping layer is used to stack the feature maps into a feature sequence; the TransformerEncoder layer is used to process the feature sequence; the global average pooling layer is used to perform global average pooling on the output of the Transformer Encoder layer; and the classification layer is used to generate the labor behavior classification result based on the output of the global average pooling layer.

[0086] In some implementations, the classification of labor activities may include tilling, sowing, weeding, spraying pesticides, or emasculating.

[0087] In some embodiments, the work behavior recognition device may further include: The modeling module is used to establish the initial model of the labor behavior recognition model, the training set of labor behavior images, and the test set of labor behavior images; The training module is used to input the training set of labor behavior images into the initial model and train the initial model using the Adam optimizer and the classification cross-entropy loss function. The testing module is used to input the test set of labor behavior images into the trained initial model for testing, and the initial model that passes the test is used as the labor behavior recognition model.

[0088] In some implementations, the modeling module can also be used for: Obtain image samples of labor activities; Data augmentation is performed on images of labor activities to obtain augmented image samples; data augmentation includes one or more of the following: brightness adjustment, rotation, horizontal flipping, and vertical flipping. A dataset consisting of labor behavior image samples and augmented image samples was established; The dataset is divided into a training set of images of labor behavior and a test set of images of labor behavior.

[0089] It should be noted that the labor behavior recognition device provided by the present invention can execute the labor behavior recognition method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0090] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 12 As shown, the electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can call logical instructions in the memory to execute a labor behavior recognition method, which includes: acquiring a sequence of labor behavior images to be recognized; inputting the labor behavior image sequence into a trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model; wherein the labor behavior recognition model is a cascaded hybrid model based on a convolutional neural network and a Transformer model.

[0091] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the labor behavior recognition method provided in the above embodiments, the method including: acquiring a sequence of labor behavior images to be recognized; inputting the labor behavior image sequence into a trained labor behavior recognition model to obtain a labor behavior classification result output by the labor behavior recognition model; wherein, the labor behavior recognition model is a cascaded hybrid model based on a convolutional neural network and a Transformer model.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the labor behavior recognition method provided in the above embodiments. The method includes: acquiring a sequence of labor behavior images to be recognized; inputting the sequence of labor behavior images into a trained labor behavior recognition model to obtain a labor behavior classification result output by the labor behavior recognition model; wherein the labor behavior recognition model is a cascaded hybrid model based on a convolutional neural network and a Transformer model.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing work behavior, characterized in that, include: Obtain the image sequence of the labor behavior to be identified; The sequence of images of labor behavior is input into a trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model. The labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

2. The labor behavior recognition method according to claim 1, characterized in that, The acquisition of the image sequence of the labor behavior to be identified includes: Acquire video data of labor activities captured by drones; Extract the image sequence of the labor activity from the video data of the labor activity.

3. The method for recognizing work behavior according to claim 1, characterized in that, The labor behavior recognition model includes an input layer, a convolutional layer, a reshaping layer, a Transformer Encoder layer, a global average pooling layer, and a classification layer. The input layer is used to input the sequence of labor behavior images; the convolutional layer uses a convolutional neural network to extract feature maps from the sequence of labor behavior images; the reshaping layer stacks the feature maps into a feature sequence; the Transformer Encoder layer processes the feature sequence; the global average pooling layer performs global average pooling on the output of the Transformer Encoder layer; and the classification layer generates a classification result of the labor behavior based on the output of the global average pooling layer.

4. The labor behavior recognition method according to claim 1, characterized in that, The classification of labor activities includes turning the soil, sowing seeds, weeding, spraying pesticides, or emasculating.

5. The method for identifying work behavior according to claim 1, characterized in that, Before inputting the sequence of labor behavior images into the trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model, the method further includes: Establish the initial model of the labor behavior recognition model, the training set of labor behavior images, and the test set of labor behavior images; The training set of the labor behavior images is input into the initial model, and the initial model is trained using the Adam optimizer and the classification cross-entropy loss function. The labor behavior image test set is input into the trained initial model for testing, and the initial model that passes the test is used as the labor behavior recognition model.

6. The labor behavior recognition method according to claim 5, characterized in that, The steps for establishing the training set and test set of the labor behavior images include: Obtain image samples of labor activities; The image samples of the labor behavior are data augmented to obtain augmented image samples; the data augmentation includes one or more of the following: brightness adjustment, rotation, horizontal flipping, and vertical flipping. A dataset consisting of the labor behavior image samples and the enhanced image samples is established; The dataset is divided into a training set of labor behavior images and a test set of labor behavior images.

7. A labor behavior recognition device, characterized in that, include: The acquisition module is used to acquire image sequences of the labor behavior to be identified; The recognition module is used to input the sequence of labor behavior images into a trained labor behavior recognition model to obtain the labor behavior classification result output by the labor behavior recognition model. The labor behavior recognition model is a cascaded hybrid model based on convolutional neural networks and Transformer models.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the labor behavior recognition method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the labor behavior recognition method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the labor behavior recognition method as described in any one of claims 1 to 6.

Images