Target behavior detection method and device, computer equipment and storage medium

Abstract

The invention relates to a target behavior detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-detected original image; inputting the original image into a target behavior detection model, detecting a head and shoulder region of a target object from the original image based on the target behavior detection model, and classifying and identifying the head and shoulder region to obtain a target detection result; wherein the target detection result comprises existence of a target behavior and absence of the target behavior; the target behavior detection model is obtained by training through a machine learning method based on a corresponding relation between a sample image and a detection result; the sample image is marked with a head and shoulder area of a sample object. According to the embodiment of the invention, the detection accuracy of the target behavior is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target behavior detection method, device, computer equipment, storage medium and computer program product. Background Art

[0002] In the act of smoking, improper handling of cigarette butts may cause fires, resulting in serious casualties and economic losses. At the same time, there are scenarios where smoking is prohibited, such as gas stations, high-speed railways, airports, hospitals, and certain enterprises and institutions. However, in the above scenarios, smoking still exists.

[0003] In the related art, smoking behavior is detected through image recognition or gas sensors, but due to environmental factors, the missed alarm rate or false alarm rate is still high. Summary of the invention

[0004] Based on this, it is necessary to provide a target behavior detection method, device, computer equipment, computer-readable storage medium and computer program product that can detect accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for detecting target behavior. The method comprises:

[0006] The detecting the head and shoulder region of the target object from the original image based on the target behavior detection model, classifying and identifying the head and shoulder region, and obtaining the target detection result includes:

[0007] Based on the target behavior detection model, detecting the head and shoulder region of the target object from the original image, classifying and identifying the head and shoulder region, and obtaining a first detection result;

[0008] Based on the target behavior detection model, detecting a palm area of ​​the target object from the image, classifying and identifying the palm area, and obtaining a second detection result;

[0009] A target detection result is obtained based on the first detection result and the second detection result.

[0010] In one embodiment, the target behavior detection model includes a backbone network, a self-attention mechanism network, and a cross-scale feature fusion network. The step of detecting the head and shoulder region of the target object from the original image based on the target behavior detection model includes:

[0011] Based on the backbone network, feature extraction is performed on the original image to obtain a first feature map, a second feature map, and a third feature map; wherein the size of the first feature map is larger than the size of the second feature map, and the size of the second feature map is larger than the size of the third feature map;

[0012] Based on the self-attention mechanism network, feature fusion is performed on the third feature map to obtain a processed third feature map;

[0013] Based on a cross-scale feature fusion network, feature fusion is performed on the first feature map, the second feature map, and the processed third feature map to obtain fused features;

[0014] Based on the fused features, the head and shoulder regions of the target object are predicted.

[0015] In one embodiment, predicting the head and shoulder region of the target object based on the fused features includes:

[0016] Based on the fused features, a corresponding initial prediction result is obtained; the initial prediction result includes a head-shoulder region and a non-head-shoulder region of the target object;

[0017] Based on the initial prediction results, the fused features are screened to obtain the screened features;

[0018] Based on the filtered features, the head and shoulders area of ​​the target object is predicted.

[0019] In one embodiment, the target behavior detection model includes a region detection model and a region classification model, the region classification model includes a first classification sub-model and a second classification sub-model, and the head and shoulder regions of the target object are detected from the original image, and the head and shoulder regions are classified and identified to obtain the target detection result, including:

[0020] Based on the region detection model, detecting the head and shoulder region and the palm region of the target object from the original image;

[0021] The classifying and identifying the head and shoulder region to obtain a target detection result includes:

[0022] Determine a detection frame of the head and shoulder region of the target object from the original image, enlarge the detection frame of the head and shoulder region, and capture an image of the head and shoulder region corresponding to the enlarged detection frame;

[0023] Inputting the head and shoulder region image into the first classification model to obtain a first detection result;

[0024] Determine a detection frame of the palm area of ​​the target object from the original image, enlarge the detection frame of the palm area image, and intercept the palm area image corresponding to the enlarged detection frame;

[0025] Inputting the palm area image into the second classification model to obtain a second detection result;

[0026] A target detection result is obtained based on the first detection result and the second detection result.

[0027] In one embodiment, the target behavior detection model is obtained by training through a machine learning method based on the correspondence between sample images and detection results, including:

[0028] Acquire a sample set; the sample set includes sample images annotated with a head and shoulder region annotation box of the sample object;

[0029] Inputting the sample image into an initial region detection model, and outputting a prediction frame of the head and shoulder region of the sample object;

[0030] Constructing a loss function based on the center distance between the annotation box and the prediction box, the diagonal length of the minimum circumscribed rectangle containing the annotation box and the prediction box, and the intersection-and-union ratio between the annotation box and the prediction box;

[0031] Based on the loss function, iteratively adjust the initial region detection model to obtain a region detection model;

[0032] Inputting the sample image into a region detection model, and outputting a prediction box of the head and shoulder region of the sample object;

[0033] Based on the prediction frame, a head and shoulder region of the sample object is intercepted to obtain a sample head and shoulder region image;

[0034] Input the sample head and shoulder area image into the initial area classification model and output the predicted category;

[0035] Based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample, the initial region classification model is iteratively adjusted to obtain a region classification model.

[0036] In a second aspect, the present application also provides a target behavior detection device. The device comprises:

[0037] An acquisition module, used for acquiring an original image to be detected;

[0038] A detection module is used to input the original image into a target behavior detection model, detect the head and shoulder area of ​​the target object from the original image based on the target behavior detection model, classify and identify the head and shoulder area, and obtain a target detection result; wherein the target detection result includes the presence of the target behavior and the absence of the target behavior; the target behavior detection model is obtained by training through a machine learning method based on the correspondence between the sample image and the detection result; the sample image is annotated with the head and shoulder area of ​​the sample object.

[0039] In one embodiment, the detection module is further used for:

[0040] Based on the target behavior detection model, detecting the head and shoulder region of the target object from the original image, classifying and identifying the head and shoulder region, and obtaining a first detection result;

[0041] Based on the target behavior detection model, detecting a palm area of ​​the target object from the image, classifying and identifying the palm area, and obtaining a second detection result;

[0042] A target detection result is obtained based on the first detection result and the second detection result.

[0043] In one embodiment, the target behavior detection model includes a backbone network, a self-attention mechanism network and a cross-scale feature fusion network, and the detection module is further used to:

[0044] Based on the backbone network, feature extraction is performed on the original image to obtain a first feature map, a second feature map, and a third feature map; wherein the size of the first feature map is larger than the size of the second feature map, and the size of the second feature map is larger than the size of the third feature map;

[0045] Based on the self-attention mechanism network, feature fusion is performed on the third feature map to obtain a processed third feature map;

[0046] Based on a cross-scale feature fusion network, feature fusion is performed on the first feature map, the second feature map, and the processed third feature map to obtain fused features;

[0047] Based on the fused features, the head and shoulder regions of the target object are predicted.

[0048] In one embodiment, the detection module is further used for:

[0049] Based on the fused features, a corresponding initial prediction result is obtained; the initial prediction result includes a head-shoulder region and a non-head-shoulder region of the target object;

[0050] Based on the initial prediction results, the fused features are screened to obtain the screened features;

[0051] Based on the filtered features, the head and shoulders area of ​​the target object is predicted.

[0052] In one embodiment, the target behavior detection model includes a region detection model and a region classification model, the region classification model includes a first classification sub-model and a second classification sub-model, and the detection module is further used to:

[0053] Based on the region detection model, detecting the head and shoulder region and the palm region of the target object from the original image;

[0054] The classifying and identifying the head and shoulder region to obtain a target detection result includes:

[0055] Determine a detection frame of the head and shoulder region of the target object from the original image, enlarge the detection frame of the head and shoulder region, and capture an image of the head and shoulder region corresponding to the enlarged detection frame;

[0056] Inputting the head and shoulder region image into the first classification model to obtain a first detection result;

[0057] Determine a detection frame of the palm area of ​​the target object from the original image, enlarge the detection frame of the palm area image, and intercept the palm area image corresponding to the enlarged detection frame;

[0058] Inputting the palm area image into the second classification model to obtain a second detection result;

[0059] A target detection result is obtained based on the first detection result and the second detection result.

[0060] In one embodiment, the device further comprises a training module, wherein the training module is configured to:

[0061] Acquire a sample set; the sample set includes sample images annotated with a head and shoulder region annotation box of the sample object;

[0062] Inputting the sample image into an initial region detection model, and outputting a prediction frame of the head and shoulder region of the sample object;

[0063] Constructing a loss function based on the center distance between the annotation box and the prediction box, the diagonal length of the minimum circumscribed rectangle containing the annotation box and the prediction box, and the intersection-and-union ratio between the annotation box and the prediction box;

[0064] Based on the loss function, iteratively adjust the initial region detection model to obtain a region detection model;

[0065] Inputting the sample image into a region detection model, and outputting a prediction box of the head and shoulder region of the sample object;

[0066] Based on the prediction frame, a head and shoulder region of the sample object is intercepted to obtain a sample head and shoulder region image;

[0067] Input the sample head and shoulder area image into the initial area classification model and output the predicted category;

[0068] Based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample, the initial region classification model is iteratively adjusted to obtain a region classification model.

[0069] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, an embodiment of the method described in any one of the embodiments of the present disclosure is implemented.

[0070] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, an embodiment of the method described in any one of the embodiments of the present disclosure is implemented.

[0071] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, an embodiment of the method described in any one of the embodiments of the present disclosure is implemented.

[0072] Compared with the traditional image recognition detection method of target behavior, such as face recognition detection or small target cigarette detection, the above-mentioned target behavior detection method, apparatus, computer equipment, storage medium and computer program product can mostly include cigarettes in the detection frame if the front face is detected; if the front face is covered by the side face or the hair, the detection frame may not include the cigarette. For example, the part of the cigarette held in the mouth often exceeds the face area by a lot, resulting in inaccurate detection results. If it is a small target cigarette detection, the cigarette belongs to the small target detection in the screen presentation. The detection difficulty of small target objects is obviously greater, and there are more missed detections and false detections. Therefore, the embodiment of the present disclosure detects the head and shoulder area of ​​the target object through the target behavior detection model, which can include the complete face area and the complete cigarette part, so that the detection accuracy of the target behavior is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of a target behavior detection method in one embodiment;

[0074] Figure 2 is a flow chart of a target behavior detection method in another embodiment;

[0075] Figure 3 is an architecture diagram of a target behavior detection model in one embodiment;

[0076] Figure 4 is a flow chart of a target behavior detection method in another embodiment;

[0077] Figure 5 is an architectural diagram of a first classification sub-model in one embodiment;

[0078] Figure 6 is an architectural diagram of a second classification sub-model in one embodiment;

[0079] Figure 7 is a flow chart of a target behavior detection method in another embodiment;

[0080] Figure 8 is a structural block diagram of a target behavior detection device in one embodiment;

[0081] Fig. 9 is an internal structure diagram of a computer device in one embodiment;

[0082] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0084] In one embodiment, Figure 1 As shown, a target behavior detection method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0085] Step S101, obtaining an original image to be detected.

[0086] The original image may include an image taken by a terminal camera, and the shooting scene may be a gas station, a gas station, a high-speed railway, an airport, a hospital, or a specific enterprise or institution.

[0087] Step S103: input the original image into a target behavior detection model, detect the head and shoulder region of the target object from the original image based on the target behavior detection model, classify and identify the head and shoulder region, and obtain a target detection result.

[0088] Wherein, the target behavior detection model is obtained by training through a machine learning method based on the correspondence between the sample image and the detection result; the sample image is annotated with the head and shoulder area of ​​the sample object. Exemplarily, the target behavior detection model can be obtained by training through a machine learning method based on a basic network, and the basic network includes but is not limited to a YOLO network architecture, an SSD network architecture, and an RT-DETR network architecture. In an exemplary embodiment, the original image can be directly input into the target behavior detection model trained by the above-mentioned basic network, and the target detection result is output. In an exemplary embodiment, a regional detection model can be trained by machine learning based on the above-mentioned basic network, the original image can be input into the regional detection model, the head and shoulder area of ​​the target object can be output, and the head and shoulder area image corresponding to the head and shoulder area can be input into the regional classification model, and the target detection result can be output. Optionally, the image of the head and shoulder area output by the regional detection model can be enlarged, for example, the magnification is fixed at 1.1 times, that is, the detection frame corresponding to the head and shoulder area is enlarged 1.1 times to the peripheral area, so as to ensure that the complete cigarette is contained as much as possible.

[0089] The head and shoulder area may be a rectangular area including both sides of the head and shoulders of the target object. Depending on the specific application scenario, the size of the rectangular area may be different, and the embodiments of the present disclosure do not impose any limitation on this.

[0090] The target detection result may include the presence of the target behavior and the absence of the target behavior. Specifically, the target behavior may include smoking behavior, that is, the target detection result may include the presence of smoking behavior and the absence of smoking behavior. Optionally, the target behavior may also include eating behavior, such as prohibiting eating and drinking in a specific exhibition tube.

[0091] In the above embodiments, compared to the traditional image recognition method of detecting target behavior, such as face recognition detection or small target cigarette detection, if the front face is detected, the cigarette can mostly be included in the detection frame; if the side face or hair covers the front face, the detection frame may not include the cigarette. For example, the part of the cigarette held in the mouth often exceeds the face area a lot, resulting in inaccurate detection results. If it is a small target cigarette detection, the cigarette is a small target detection in the screen presentation. The detection difficulty of small target objects is obviously greater, and there are more missed detections and false detections. Therefore, the embodiment of the present disclosure detects the head and shoulder area of ​​the target object through the target behavior detection model, which can include the complete face area and the complete cigarette part, so that the detection accuracy of the target behavior is higher.

[0092] In one embodiment, based on the target behavior detection model, detecting the head and shoulder region of the target object from the original image, classifying and identifying the head and shoulder region, and obtaining the target detection result includes:

[0093] Based on the target behavior detection model, the head and shoulder region of the target object is detected from the original image, and the head and shoulder region is classified and identified to obtain a first detection result.

[0094] Based on the target behavior detection model, a palm area of ​​the target object is detected from the image, and the palm area is classified and identified to obtain a second detection result.

[0095] A target detection result is obtained based on the first detection result and the second detection result.

[0096] In the embodiment of the present disclosure, the target behavior detection model can be obtained by training in the manner of the above embodiment, and the present disclosure will not repeat it here. In an exemplary embodiment, based on the target behavior detection model, the head and shoulder area of ​​the target object is detected from the original image. Optionally, the head and shoulder area can be classified and identified by a classification model corresponding to the head and shoulder area to obtain a first detection result. Among them, the first detection result may include the presence of smoking behavior or the absence of smoking behavior.

[0097] In an exemplary embodiment, based on the target behavior detection model, the palm area of ​​the target object is detected from the original image. Optionally, the palm area can be classified and identified by a classification model corresponding to the palm area to obtain a second detection result. The second detection result may include holding a cigarette or not holding a cigarette.

[0098] In the embodiment of the present disclosure, if the first detection result detects the presence of smoking behavior and the second detection result detects that the hand is not holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the absence of smoking behavior and the second detection result detects that the hand is holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the presence of smoking behavior and the second detection result detects that the hand is holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the absence of smoking behavior and the second detection result detects that the hand is not holding a cigarette, then the detection result may be the absence of smoking behavior.

[0099] The above-mentioned embodiment replaces the cigarette detection in traditional image recognition by detecting the head and shoulder area and the palm area. The detection of the head and shoulder and the palm mainly takes into account the situation that smoking may include holding the cigarette in the hands and holding the cigarette in the mouth. The simple cigarette body target detection is transformed into the joint classification and recognition of the head and shoulder area and the palm area, which can avoid many problems caused by traditional image recognition detection and further improve the accuracy of the detection results.

[0100] In one embodiment, reference Figure 2As shown, the target behavior detection model includes a backbone network, a self-attention mechanism network and a cross-scale feature fusion network. The target behavior detection model is based on which the head and shoulder area of ​​the target object is detected from the original image, including:

[0101] Step S201, based on the backbone network, extract features of the original image to obtain a first feature map, a second feature map and a third feature map; wherein the size of the first feature map is larger than the size of the second feature map, and the size of the second feature map is larger than the size of the third feature map.

[0102] The backbone network may include a convolutional neural network, such as Resnet50. Different convolution steps are set to sample feature maps of various scales. For example, the downsampling multiples are set to 8, 16, and 32, respectively. Assuming that the size of the original image is 640×640, the first feature map after downsampling is 80×80, 40×40, and 20×20. In an exemplary embodiment, reference Figure 3 As shown, the original image 301 is input into the backbone network, and after feature extraction, a first feature map S3, a second feature map S4 and a third feature map S5 are obtained.

[0103] Step S203: Based on the self-attention mechanism network, feature fusion is performed on the third feature map to obtain a processed third feature map.

[0104] Specifically, the Flatten function can be used to stretch the third feature map S5 into a vector form, and the stretched vector is adjusted back to the original size of the third feature map S5 through the self-attention mechanism network to form the F5 feature. Figure 3 As shown in the figure, the self-attention mechanism network AIFI (Attention-based Intra-scale Feature Interaction) can be constructed by standard multi-head self-attention (MSA) and feed-forward neural network (FFN).

[0105] Step S205: Based on a cross-scale feature fusion network, feature fusion is performed on the first feature map, the second feature map, and the processed third feature map to obtain fused features.

[0106] Based on a cross-scale feature fusion network, such as CCFF (CNN-based Cross-scale Feature Fusion), the first feature map S3, the second feature map S4, and the processed third feature map F5 are fused to obtain fused features. The CCFF network can be constructed by the Panoptic Feature Pyramid Network (PaFPN). The PaFPN structure first uses several layers of 1x1 convolution to map the number of channels of all features to the same number (such as 256), and then fuses the top-down and bottom-up features to obtain the final fused features. Specifically, multiple 1x1 convolutions are first used to map the number of channels of feature maps of different layers from the main network (such as ResNet, FPN, etc.) to the same number (such as 256). The purpose of this step is to unify the feature dimensions of different layers, making subsequent feature fusion more efficient and convenient. Feature Fusion Process The feature fusion process of PaFPN is mainly divided into two directions: bottom-up and top-down. Among them, Bottom-Up feature fusion bottom-up path: In this process, PaFPN first fuses from the bottom layer (low-resolution feature map) upward. By gradually integrating the low-level resolution features into the high-level feature map, it is easy to capture fine-grained information. Horizontal connection: Each time a combination is made, a horizontal connection is made to fuse the high-level features with the corresponding low-level features to enhance the richness of the features. Top-down path: Next, start downsampling and fusion of features from the top-level features. This step helps to improve the resolution of the feature map and retain greater contextual information during the fusion process.

[0107] Step S207: predicting the head and shoulder region of the target object based on the fused features.

[0108] In the embodiment of the present disclosure, based on the fused features, the features are classified at the pixel level to obtain the head and shoulder area and the non-head and shoulder area of ​​the target object. The head and shoulder area of ​​the target object can be marked by a detection frame.

[0109] In the above embodiment, through efficient feature extraction, a higher quality prediction box can be generated, and the use of the self-attention mechanism can better capture the relationship between targets and achieve accurate positioning, thereby reducing non-maximum suppression (NMS) and improving detection efficiency and accuracy.

[0110] In one embodiment, predicting the head and shoulder region of the target object based on the fused features includes:

[0111] Based on the fused features, a corresponding initial prediction result is obtained; the initial prediction result includes a head-shoulder region and a non-head-shoulder region of the target object.

[0112] Based on the initial prediction results, the fused features are screened to obtain screened features.

[0113] Based on the filtered features, the head and shoulders area of ​​the target object is predicted.

[0114] Specifically, for example, the initial prediction result is [8400, 2], which indicates 8400 prediction results, each of which includes 2 categories. By selecting the category with the largest score, 8400 1-dimensional prediction results can be obtained. Exemplarily, the K features with the highest scores, such as the 300 features with the highest scores, can be selected as the filtered features. Based on the filtered features, the head and shoulder area of ​​the target object is predicted.

[0115] Through the above embodiment, features with higher scores are screened out, and the results have higher certainty and reliability, which can minimize the uncertainty of the prediction results. Based on this explicit optimization uncertainty, a high-quality initial query is provided, thereby improving the accuracy of head and shoulder region detection.

[0116] In one embodiment, reference Figure 4 As shown, the target behavior detection model includes a region detection model and a region classification model, the region classification model includes a first classification sub-model and a second classification sub-model, the head and shoulder region of the target object is detected from the original image, the head and shoulder region is classified and identified, and the target detection result is obtained, including:

[0117] Step S401: Detect the head and shoulder area and the palm area of ​​the target object from the original image based on the area detection model.

[0118] Specifically, a region detection model can be obtained by training through a machine learning method based on a basic network, wherein the basic network includes but is not limited to a YOLO network architecture, an SSD network architecture, and an RT-DETR network architecture, etc. In an exemplary embodiment, the head and shoulder region and the palm region of the target object are detected from the original image through the region detection model.

[0119] The classifying and identifying the head and shoulder region to obtain a target detection result includes:

[0120] Step S403, determining a detection frame of the head and shoulder region of the target object from the original image, enlarging the detection frame of the head and shoulder region, and capturing an image of the head and shoulder region corresponding to the enlarged detection frame.

[0121] In an exemplary embodiment, a detection frame of the head and shoulder region of the target object is determined from the original image, and the detection frame can be enlarged by a fixed factor, for example, 1.1 times, and an image of the head and shoulder region corresponding to the enlarged detection frame is captured.

[0122] Step S405: input the head and shoulder region image into the first classification model to obtain a first detection result.

[0123] In an exemplary embodiment, the first classification model can be trained by deep learning based on the VanillaNet network. VanillaNet is a neural network architecture that focuses on minimalism and efficiency. It has a simple design and a small number of layers, avoiding complex operations such as deep architecture and self-attention. The key features of VanillaNet include a deep training strategy, which initially uses an activation function to train two convolutional layers, and then the activation function is gradually simplified to an identity mapping, allowing layer merging. In addition, VanillaNet also uses parallel stacked activation functions to improve nonlinearity, thereby improving the performance of simple networks. Among them, the first detection result may include the presence of target behavior and the absence of target behavior.

[0124] In the embodiments of the present disclosure, reference Figure 5 As shown in the figure, the VanillaNet-5 structure is the backbone of the VanillaNet network with a depth of 5. When the input image is 112x112 and the number of classification categories is 2, the original VanillaNet-5 structure can be appropriately adjusted in terms of input, output, and number of channels.

[0125] Step S407, determining a detection frame of the palm area of ​​the target object from the original image, enlarging the detection frame of the palm area image, and capturing the palm area image corresponding to the enlarged detection frame.

[0126] In an exemplary embodiment, a detection frame of the palm area of ​​the target object is determined from the original image, and the detection frame can be enlarged by a fixed factor, such as 1.1 times, and the palm area image corresponding to the enlarged detection frame is captured.

[0127] Step S409: input the palm area image into the second classification model to obtain a second detection result.

[0128] In an exemplary embodiment, the second classification model can be based on the VanillaNet network and trained by deep learning. The second detection result can include the hand holding the cigarette and the hand not holding the cigarette. Figure 6As shown in the figure, the VanillaNet-5 structure is the backbone of the VanillaNet network with a depth of 5. Considering that the palm area image is smaller, when the input image is 56x56 and the number of classification categories is 2, the original VanillaNet-5 structure can be appropriately adjusted in terms of input, output, and number of channels.

[0129] Step S411, obtaining a target detection result based on the first detection result and the second detection result.

[0130] In the embodiment of the present disclosure, if the first detection result detects the presence of smoking behavior and the second detection result detects that the hand is not holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the absence of smoking behavior and the second detection result detects that the hand is holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the presence of smoking behavior and the second detection result detects that the hand is holding a cigarette, then the detection result may be the presence of smoking behavior; if the first detection result detects the absence of smoking behavior and the second detection result detects that the hand is not holding a cigarette, then the detection result may be the absence of smoking behavior.

[0131] In the above embodiment, the target behavior detection model includes a region detection model and a region classification model, and the region classification model includes a first classification sub-model and a second classification sub-model. Compared with the traditional one-hot encoding method of the target frame category, it is possible that when the positioning of the target frame is not accurate enough, the category has already been learned in advance, which is "misaligned", resulting in the inability to more accurately predict the positioning of the current target frame. In the disclosed embodiment, an independent classifier is used after target detection to improve the accuracy of the detection results.

[0132] In one embodiment, reference Figure 7 As shown, the target behavior detection model is obtained by training based on the correspondence between sample images and detection results through machine learning methods, including:

[0133] Step S701, obtaining a sample set; the sample set includes sample images annotated with a head and shoulder region annotation box of a sample object.

[0134] The sample images may be collected through the Internet or other sources, for example, camera surveillance images for a specified deployment scenario may be collected. Optionally, the sample images may be preprocessed, such as image mean variance processing, mirror flipping, random erasure, random noise, and the like.

[0135] Step S703: input the sample image into an initial region detection model, and output a prediction box of the head and shoulder region of the sample object.

[0136] The initial region detection model may include a model trained by a machine learning method based on a basic network, wherein the basic network includes but is not limited to a YOLO network architecture, an SSD network architecture, and an RT-DETR network architecture. In an exemplary embodiment, the sample image is input into the initial region detection model, and a prediction box of the head and shoulder region of the sample object is output. In another exemplary embodiment, the initial region detection model may detect the palm region at the same time, that is, the sample image is input into the initial region detection model, and a prediction box of the palm region of the sample object is output.

[0137] Step S705 , constructing a loss function based on the center distance between the labeled box and the predicted box, the diagonal length of the minimum circumscribed rectangle containing the labeled box and the predicted box, and the intersection-and-union ratio between the labeled box and the predicted box.

[0138] Specifically, the prediction box is expressed as , the annotation box is represented by , the center distance between the annotation box and the prediction box is expressed as , the diagonal length of the minimum bounding rectangle containing the annotation box and the prediction box can be expressed as , the intersection of the annotation box and the predicted box , which is the area of ​​the intersection of the two boxes divided by the area of ​​the union of the two boxes. In an exemplary embodiment, the loss function It can be expressed as follows (1)

[0139] (1)

[0140] In an exemplary embodiment, the uncertainty factor can also be considered to construct the loss function. Specifically, for each encoder feature calculator classification distribution and positioning distribution , uncertainty factor Defined as the difference between the classification distribution and the location distribution, the uncertainty factor It is expressed as:

[0141] (2)

[0142] in, Represents the encoder characteristics, and the uncertainty factor Introduced into the loss function, the intermediate loss function is expressed as follows:

[0143] (3)

[0144] The annotation box is represented by , the prediction box is represented as , the annotation category is represented as , the predicted category is expressed as In the disclosed embodiment, the prediction content includes a prediction box and a prediction category. The prediction box generally takes the values ​​of the upper left and lower right points of the rectangular box. The prediction category can be 0 or 1, where 0 represents the head and shoulder area, and 1 represents the non-head and shoulder area.

[0145] In an exemplary embodiment, the total loss function can be expressed as:

[0146] (4)

[0147] in, , , is the weight coefficient of each type of loss.

[0148] Step S707: iteratively adjust the initial region detection model based on the loss function to obtain a region detection model.

[0149] Specifically, the loss can be calculated according to the above loss function, and based on the loss minimization strategy, the network parameters of the initial region detection model are adjusted to obtain the region detection model.

[0150] Step S709: input the sample image into a region detection model, and output a prediction box of the head and shoulder region of the sample object.

[0151] In an exemplary embodiment, the sample image is input into a region detection model, and a prediction box of the head and shoulder region of the sample object is output. Alternatively, the sample image is input into a region detection model, and a prediction box of the palm region of the sample object is output.

[0152] Step S711: based on the prediction frame, the head and shoulder region of the sample object is clipped to obtain a sample head and shoulder region image.

[0153] For the prediction box of the head and shoulder area, the head and shoulder area of ​​the sample object is cut out to obtain a sample head and shoulder area image. Optionally, the prediction box can be randomly enlarged, for example, by 1.0 to 1.2 times. Based on the enlarged prediction box, the head and shoulder area of ​​the sample object is cut out to obtain a sample head and shoulder area image. In an exemplary embodiment, the sample head and shoulder area image inherits the category of the original annotation box.

[0154] It is understandable that the palm region of the sample object may be clipped for the predicted frame of the palm region to obtain a sample palm region image. Similarly, the sample palm region image inherits the category of the original annotation frame.

[0155] Step S713, input the sample head and shoulder region image into the initial region classification model, and output the predicted category.

[0156] In an exemplary embodiment, the sample head and shoulder region image is input into the initial region classification model, and the predicted category is output. In another exemplary embodiment, the initial region classification model may include a first classification sub-model and a second classification sub-model. Thus, the sample head and shoulder region image is input into the first classification sub-model to obtain the predicted category of the head and shoulder region, and the sample palm region image is input into the second classification sub-model to obtain the predicted category of the palm region.

[0157] Step S715, based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample, iteratively adjust the initial region classification model to obtain a region classification model.

[0158] In an exemplary embodiment, the initial regional classification model is iteratively adjusted based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample to obtain a regional classification model. In another exemplary embodiment, the initial first classification sub-model can be iteratively adjusted based on the difference between the predicted category and the labeled category of the head and shoulder region to obtain a first classification sub-model. Based on the difference between the predicted category and the labeled category of the palm region, the initial second classification sub-model is iteratively adjusted to obtain a second classification sub-model.

[0159] In the above embodiment, a loss function is constructed based on the center distance between the annotation box and the prediction box, the diagonal length of the minimum circumscribed rectangle containing the annotation box and the prediction box, and the intersection-over-union ratio between the annotation box and the prediction box. Compared with the traditional DIOU calculation method, the disclosed embodiment is more in line with the target box regression mechanism, and the distance, overlap rate and scale between the target box and the prediction box are taken into account, making the regression of the target box more stable. Furthermore, when formula (4) is used as the loss function, the larger the IOU between the prediction box and the original annotation box, Loss and The smaller the loss is, the smaller the model will be optimized in the direction of smaller loss during training iterations, so that the prediction box result is predicted in the direction of higher prediction. The corresponding prediction box is made to develop in the direction of high score. As a result, the final constraint effect is presented as "features with high IoU produce high classification scores, and features with low IOU produce low classification scores". Here, IOU is a general term for the intersection-over-union ratio of the target box, which achieves a better effect.

[0160] In the above embodiment, during training, the detector is constrained to generate high classification scores for features of detection boxes with high IoU ratios and low classification scores for features of detection boxes with low IoU ratios. As a result, the prediction boxes corresponding to the features with higher scores selected by the model according to the classification scores have both high classification scores and high IoU ratio scores. Finally, the object query is iteratively optimized through the decoder (Decoder & Head) with an auxiliary prediction head to generate categories and bounding boxes.

[0161] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0162] Based on the same inventive concept, the embodiment of the present application also provides a target behavior detection device for implementing the target behavior detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target behavior detection device embodiments provided below can refer to the limitations of the target behavior detection method above, and will not be repeated here.

[0163] In one embodiment, Figure 8 As shown, a target behavior detection device 800 is provided, comprising:

[0164] The acquisition module 801 is used to acquire the original image to be detected;

[0165] The detection module 803 is used to input the original image into the target behavior detection model, detect the head and shoulder area of ​​the target object from the original image based on the target behavior detection model, classify and identify the head and shoulder area, and obtain the target detection result; wherein the target detection result includes the presence of the target behavior and the absence of the target behavior; the target behavior detection model is obtained by training through a machine learning method based on the correspondence between the sample image and the detection result; the sample image is annotated with the head and shoulder area of ​​the sample object.

[0166] In one embodiment, the detection module is further used for:

[0167] Based on the target behavior detection model, detecting the head and shoulder region of the target object from the original image, classifying and identifying the head and shoulder region, and obtaining a first detection result;

[0168] Based on the target behavior detection model, detecting a palm area of ​​the target object from the image, classifying and identifying the palm area, and obtaining a second detection result;

[0169] A target detection result is obtained based on the first detection result and the second detection result.

[0170] In one embodiment, the target behavior detection model includes a backbone network, a self-attention mechanism network and a cross-scale feature fusion network, and the detection module is further used to:

[0171] Based on the backbone network, feature extraction is performed on the original image to obtain a first feature map, a second feature map, and a third feature map; wherein the size of the first feature map is larger than the size of the second feature map, and the size of the second feature map is larger than the size of the third feature map;

[0172] Based on the self-attention mechanism network, feature fusion is performed on the third feature map to obtain a processed third feature map;

[0173] Based on a cross-scale feature fusion network, feature fusion is performed on the first feature map, the second feature map, and the processed third feature map to obtain fused features;

[0174] Based on the fused features, the head and shoulder regions of the target object are predicted.

[0175] In one embodiment, the detection module is further used for:

[0176] Based on the fused features, a corresponding initial prediction result is obtained; the initial prediction result includes a head-shoulder region and a non-head-shoulder region of the target object;

[0177] Based on the initial prediction results, the fused features are screened to obtain the screened features;

[0178] Based on the filtered features, the head and shoulders area of ​​the target object is predicted.

[0179] In one embodiment, the target behavior detection model includes a region detection model and a region classification model, the region classification model includes a first classification sub-model and a second classification sub-model, and the detection module is further used to:

[0180] Based on the region detection model, detecting the head and shoulder region and the palm region of the target object from the original image;

[0181] The classifying and identifying the head and shoulder region to obtain a target detection result includes:

[0182] Determine a detection frame of the head and shoulder region of the target object from the original image, enlarge the detection frame of the head and shoulder region, and capture an image of the head and shoulder region corresponding to the enlarged detection frame;

[0183] Inputting the head and shoulder region image into the first classification model to obtain a first detection result;

[0184] Determine a detection frame of the palm area of ​​the target object from the original image, enlarge the detection frame of the palm area image, and intercept the palm area image corresponding to the enlarged detection frame;

[0185] Inputting the palm area image into the second classification model to obtain a second detection result;

[0186] A target detection result is obtained based on the first detection result and the second detection result.

[0187] In one embodiment, the device further comprises a training module, wherein the training module is configured to:

[0188] Acquire a sample set; the sample set includes sample images annotated with a head and shoulder region annotation box of the sample object;

[0189] Inputting the sample image into an initial region detection model, and outputting a prediction frame of the head and shoulder region of the sample object;

[0190] Constructing a loss function based on the center distance between the annotation box and the prediction box, the diagonal length of the minimum circumscribed rectangle containing the annotation box and the prediction box, and the intersection-and-union ratio between the annotation box and the prediction box;

[0191] Based on the loss function, iteratively adjust the initial region detection model to obtain a region detection model;

[0192] Inputting the sample image into a region detection model, and outputting a prediction box of the head and shoulder region of the sample object;

[0193] Based on the prediction frame, a head and shoulder region of the sample object is intercepted to obtain a sample head and shoulder region image;

[0194] Input the sample head and shoulder area image into the initial area classification model and output the predicted category;

[0195] Based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample, the initial region classification model is iteratively adjusted to obtain a region classification model.

[0196] Each module in the above target behavior detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0197] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store target behavior detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target behavior detection method is implemented.

[0198] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a target behavior detection method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0199] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0201] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0202] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target behavior detection method, characterized in that: The method comprises: Obtain the original image to be detected; The original image is input into a target behavior detection model. Based on the target behavior detection model, the head and shoulder regions of the target object are detected from the original image, and the head and shoulder regions are classified and identified to obtain a target detection result; wherein the target detection result includes the presence and absence of target behavior; the target behavior detection model is obtained by training through a machine learning method based on the correspondence between the sample image and the detection result; the sample image is annotated with the head and shoulder regions of the sample object.

2. The method according to claim 1, characterized in that The detecting the head and shoulder region of the target object from the original image based on the target behavior detection model, classifying and identifying the head and shoulder region, and obtaining the target detection result includes: Based on the target behavior detection model, detecting the head and shoulder region of the target object from the original image, classifying and identifying the head and shoulder region, and obtaining a first detection result; Based on the target behavior detection model, detecting a palm area of ​​the target object from the image, classifying and identifying the palm area, and obtaining a second detection result; A target detection result is obtained based on the first detection result and the second detection result.

3. The method according to claim 1, characterized in that The target behavior detection model includes a backbone network, a self-attention mechanism network, and a cross-scale feature fusion network. The detecting the head and shoulder area of ​​the target object from the original image based on the target behavior detection model includes: Based on the backbone network, feature extraction is performed on the original image to obtain a first feature map, a second feature map, and a third feature map; wherein the size of the first feature map is larger than the size of the second feature map, and the size of the second feature map is larger than the size of the third feature map; Based on the self-attention mechanism network, feature fusion is performed on the third feature map to obtain a processed third feature map; Based on a cross-scale feature fusion network, feature fusion is performed on the first feature map, the second feature map, and the processed third feature map to obtain fused features; Based on the fused features, the head and shoulder regions of the target object are predicted.

4. The method according to claim 3, characterized in that The method of predicting the head and shoulder region of the target object based on the fused features includes: Based on the fused features, a corresponding initial prediction result is obtained; the initial prediction result includes a head-shoulder region and a non-head-shoulder region of the target object; Based on the initial prediction results, the fused features are screened to obtain the screened features; Based on the filtered features, the head and shoulders area of ​​the target object is predicted.

5. The method according to claim 1, characterized in that The target behavior detection model includes a region detection model and a region classification model, the region classification model includes a first classification sub-model and a second classification sub-model, the head and shoulder region of the target object is detected from the original image, the head and shoulder region is classified and identified, and the target detection result is obtained, including: Based on the region detection model, detecting the head and shoulder region and the palm region of the target object from the original image; The classifying and identifying the head and shoulder region to obtain a target detection result includes: Determine a detection frame of the head and shoulder region of the target object from the original image, enlarge the detection frame of the head and shoulder region, and capture an image of the head and shoulder region corresponding to the enlarged detection frame; Inputting the head and shoulder region image into the first classification model to obtain a first detection result; Determine a detection frame of the palm area of ​​the target object from the original image, enlarge the detection frame of the palm area image, and capture the palm area image corresponding to the enlarged detection frame; Inputting the palm area image into the second classification model to obtain a second detection result; A target detection result is obtained based on the first detection result and the second detection result.

6. The method according to any one of claims 1 to 5, characterized in that The target behavior detection model is based on the correspondence between sample images and detection results, and is obtained through training using a machine learning method, including: Acquire a sample set; the sample set includes sample images annotated with a head and shoulder region annotation box of the sample object; Inputting the sample image into an initial region detection model, and outputting a prediction frame of the head and shoulder region of the sample object; Constructing a loss function based on the center distance between the marked box and the predicted box, the diagonal length of the minimum circumscribed rectangle containing the marked box and the predicted box, and the intersection-and-union ratio between the marked box and the predicted box; Based on the loss function, iteratively adjust the initial region detection model to obtain a region detection model; Inputting the sample image into a region detection model, and outputting a prediction box of the head and shoulder region of the sample object; Based on the prediction frame, a head and shoulder region of the sample object is intercepted to obtain a sample head and shoulder region image; Input the sample head and shoulder area image into the initial area classification model and output the predicted category; Based on the difference between the predicted category and the labeled category corresponding to the head and shoulder region of the sample, the initial region classification model is iteratively adjusted to obtain a region classification model.

7. A target behavior detection device, characterized in that: The device comprises: An acquisition module, used for acquiring an original image to be detected; A detection module is used to input the original image into a target behavior detection model, detect the head and shoulder area of ​​the target object from the original image based on the target behavior detection model, classify and identify the head and shoulder area, and obtain a target detection result; wherein the target detection result includes the presence of the target behavior and the absence of the target behavior; the target behavior detection model is obtained by training through a machine learning method based on the correspondence between the sample image and the detection result; the sample image is annotated with the head and shoulder area of ​​the sample object.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images