Behavior recognition method and behavior recognition device

Abstract

The present disclosure proposes a behavior recognition method and a behavior recognition device, which relate to the field of computer vision. By detecting and tracking human bodies and bags with strap labels on surveillance videos, a sequence of images of human bodies and bags, a sequence of human body images, and a sequence of coordinates of human bodies and bags are obtained. The spatiotemporal interaction features of human bodies and bags are determined based on the sequence of images of human bodies and bags, the spatiotemporal posture features of human bodies are determined based on the sequence of human body images, and the spatiotemporal mask features of human bodies and bags are determined based on the sequence of coordinates of human bodies and bags. These features are fused and classified to obtain the recognition results of human bodies breaking the strap labels of bags. In this way, the human bodies breaking the strap labels of bags can be accurately identified, risks can be warned, and the customs can be assisted to accurately and efficiently inspect and supervise inbound and outbound articles.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a behavior recognition method and a behavior recognition device. Background Art

[0002] Customs is responsible for supervising the inbound and outbound means of transport, goods, luggage, mail and other items, including the inspection of inbound and outbound items, ensuring that all items comply with the requirements of relevant laws and regulations, and preventing the entry and exit of contraband or illegal items. With the development of economic globalization, it is urgent to improve the quality and efficiency of customs supervision.

[0003] In order to improve the efficiency of supervision, some customs have adopted the pre-inspection mode, which pre-scans the luggage to determine the different risk levels, and marks the risky luggage with special straps. Customs only needs to conduct key inspections on the luggage with straps near the exit, while luggage without straps can be quickly cleared, which greatly improves the inspection speed and enhances the customs clearance experience of passengers. However, in practice, it is found that some passengers deliberately remove the risk-marking straps to evade inspection. Summary of the invention

[0004] In the disclosed embodiment, the monitoring video is used to detect and track the human body and the luggage with the strap mark, so as to obtain the human body and the luggage image sequence, the human body image sequence, and the human body and the luggage coordinate sequence, and the human body and the luggage temporal and spatial interaction characteristics are determined based on the human body and the luggage image sequence, the human body temporal and spatial posture characteristics are determined based on the human body image sequence, and the human body and the luggage coordinate sequence is determined. These features are fused and classified to obtain the recognition result of the human body's breaking behavior of the luggage strap mark. In this way, the human body's breaking behavior of the luggage strap mark is accurately identified, the risk is warned, and the customs is assisted to accurately and efficiently inspect and supervise inbound and outbound articles.

[0005] Some embodiments of the present disclosure propose a behavior recognition method, including: detecting and tracking human bodies and bags with strap markings on surveillance videos to obtain a sequence of images of human bodies and bags, a sequence of human body images, and a sequence of human body and bags coordinates; processing the sequence of human body and bags image to determine a first feature characterizing the spatiotemporal interaction characteristics of human bodies and bags; processing the sequence of human body images to determine a second feature characterizing the spatiotemporal posture characteristics of human bodies; processing the sequence of human body and bags coordinates to determine a third feature characterizing the spatiotemporal mask characteristics of human bodies and bags; fusing and classifying the first feature, the second feature, and the third feature to obtain a recognition result of the human body's breaking behavior of the bags' strap markings.

[0006] In some embodiments, detecting a human body and a bag with a bag strap mark in a surveillance video includes: performing target detection on each frame of the surveillance video, and outputting a detection result in which a human body and a bag with a bag strap mark are detected simultaneously in one frame of the image.

[0007] In some embodiments, tracking a human body and a bag with a bag strap mark in a surveillance video includes: tracking images in which the human body and the bag with a bag strap mark intersect in the detection results.

[0008] In some embodiments, obtaining a human body and luggage pair picture sequence, a human body picture sequence, and a human body and luggage pair coordinate sequence includes: based on multiple frames of images with the same human body and luggage pair obtained by detection and tracking, combining the detection frame information of the human body and the detection frame information of the luggage in each frame of the image, determining the human body and luggage pair picture sequence, the human body and luggage pair coordinate sequence, and combining the detection frame information of the human body in each frame of the image to determine the human body picture sequence.

[0009] In some embodiments, processing a sequence of images of human bodies and luggage pairs includes: extracting visual features of each image of human bodies and luggage pairs in the sequence of images of human bodies and luggage pairs; and performing average pooling processing on the visual features of each image of human bodies and luggage pairs to obtain the first feature.

[0010] In some embodiments, processing a human body picture sequence includes: detecting each human body key point of each human body picture in the human body picture sequence, and connecting each human body key point to obtain a human body posture map; encoding each human body picture based on the human body posture map of each human body picture to obtain a human body posture feature map; performing convolution processing and spatial maximum pooling processing on each human body posture feature map to extract human body spatial posture features; performing time average pooling processing on each human body spatial posture feature to obtain the second feature.

[0011] In some embodiments, encoding each human body image based on the human body posture graph of each human body image includes: encoding the pixel values ​​on the lines connecting different parts in the human body posture graph of each human body image as different values ​​in a first group of values; encoding the pixel values ​​outside the lines connecting each part in the human body posture graph of each human body image as a second value.

[0012] In some embodiments, processing the coordinate sequence of human body and luggage pairs includes: generating spatial masks of the human body, luggage, and the intersection of the human body and luggage in the circumscribed rectangular frame of each human body and luggage pair in the coordinate sequence of human body and luggage pairs; encoding the circumscribed rectangular frame of each human body and luggage pair based on the spatial mask in the circumscribed rectangular frame of each human body and luggage pair to obtain a spatial mask code; performing convolution processing and spatial maximum pooling processing on each spatial mask code to extract spatial mask features; performing temporal average pooling processing on each spatial mask feature to obtain the third feature.

[0013] In some embodiments, encoding the bounding rectangle of each human body and luggage pair includes: encoding the value within the space mask in the bounding rectangle of each human body and luggage pair as a third value; encoding the value outside the space mask in the bounding rectangle of each human body and luggage pair as a fourth value.

[0014] In some embodiments, the first feature, the second feature, and the third feature are fused and classified to obtain an identification result of a human body's behavior of breaking open the label on the strap of a bag, including: concatenating the first feature and the second feature, fusing the features through a fully connected network, and inputting the fused features into a classification function to obtain a first classification result; processing the third feature through the fully connected network and the classification function in sequence to obtain a second classification result; and fusing the first classification result and the second classification result to obtain an identification result of a human body's behavior of breaking open the label on the strap of a bag.

[0015] In some embodiments, the method further includes: in the process of training a behavior recognition model for implementing the behavior recognition method, updating the parameters of the behavior recognition model according to the gap information between the predicted value and the reference value of the recognition result of the demolition behavior output by the behavior recognition model.

[0016] Some embodiments of the present disclosure provide a behavior recognition device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a behavior recognition method based on instructions stored in the memory.

[0017] Some embodiments of the present disclosure propose a behavior recognition device, including: a detection and tracking unit, configured to detect and track a human body and a bag with a strap mark on a surveillance video to obtain a human body and bag pair picture sequence, a human body picture sequence, and a human body and bag pair coordinate sequence; a behavior recognition unit, configured to process the human body and bag pair picture sequence to determine a first feature characterizing the spatiotemporal interaction feature of the human body and the bag; process the human body picture sequence to determine a second feature characterizing the spatiotemporal posture feature of the human body; process the human body and bag pair coordinate sequence to determine a third feature characterizing the spatiotemporal mask feature of the human body and the bag; fuse and classify the first feature, the second feature, and the third feature to obtain a recognition result of the human body's breaking behavior of the bag strap mark.

[0018] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which implements the steps of a behavior recognition method when executed by a processor.

[0019] Some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the steps of the behavior recognition method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0021] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0022] Figure 1 A schematic diagram showing a behavior recognition method according to some embodiments of the present disclosure.

[0023] Figure 2 A schematic diagram showing some embodiments of the present disclosure using a human body's spatiotemporal interactive behavior recognition network to detect whether a human body has engaged in abnormal behavior of tampering with a bag's strap.

[0024] Figure 3 A schematic diagram showing a behavior recognition device according to some embodiments of the present disclosure.

[0025] Figure 4 A schematic diagram showing a behavior recognition device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0027] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0028] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0029] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0030] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0031] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0032] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0035] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0036] In addition, in order to avoid obscuring the present disclosure due to unnecessary details, only the processing steps and / or device structures closely related to at least the scheme according to the present disclosure are shown in the drawings, and other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0037] Passengers' removal of luggage strap labels is interactive. During the removal process, human actions and luggage status are continuously changing. Without context, it is difficult to make accurate judgments. In addition, airports and other customs ports are densely populated with a large flow of people. The removal process not only needs to pay attention to human actions, but also needs to pay attention to the interaction between people and luggage, changes in luggage status, etc., which puts a lot of pressure on the system.

[0038] In response to the needs of the early machine inspection mode, the disclosed embodiment proposes a recognition scheme for the bag strap labels of passengers who have opened their luggage based on the fusion of spatiotemporal multi-features. The video data of the surveillance camera is used in combination with target detection, posture estimation and other technical means to locate the suspect and the suspected bag in real time. By integrating the time and space features of the passenger's movements and postures, and the characteristic changes between the passenger and the bag, the passenger who is opening the bag strap labels of the luggage is discovered in time, and a prompt is sent to the customs staff, so that the abnormal state of the suspected bag and the abnormal behavior of the suspect can be discovered in time, avoiding the loss of information such as the location state of the suspected bag, and assisting the customs to accurately and efficiently inspect and supervise inbound and outbound articles.

[0039] The disclosed embodiments utilize target detection algorithms and target tracking algorithms to identify and locate instance information in images, such as the location and category information of passengers and bags with bag strap labels, etc. Figure 1 Step 110 in the above; then input the trained human body's spatiotemporal interaction behavior recognition network to detect whether the human body has an abnormal behavior of breaking the bag's strap mark, see Figure 1 and Figure 2 Steps 120-150 in the above method. Based on the changes in human posture characteristics, the method pays attention to the characteristics of the intersection area between people and bags, and integrates the information of time and space dimensions. By integrating various characteristics, the accuracy of the recognition of the breaking and unlocking behavior is increased.

[0040] Figure 1 A schematic diagram showing a behavior recognition method according to some embodiments of the present disclosure.

[0041] like Figure 1 As shown, the behavior recognition method of this embodiment includes steps 110-150.

[0042] In step 110, the monitoring video is used to detect and track a human body and a bag with a bag strap mark, so as to obtain a human body and bag pair image sequence, a human body image sequence, and a human body and bag pair coordinate sequence.

[0043] Through camera monitoring, the live video stream data collected by the surveillance camera is input into the target detection algorithm to detect instances in each frame of the image. The detected categories include human body, bags with strap labels, and ordinary bags without strap labels.

[0044] The object detection algorithm solves the classification problem of "what" and the positioning problem of "where". Specifically, the object detection algorithm automatically identifies the category of the target object in a given image or video, and marks the location of the target object in the image (usually represented by a bounding box, also called a detection box).

[0045] Object detection, for example, includes the following steps.

[0046] (1) Feature extraction: Extract features from images through deep learning models such as convolutional neural networks (CNN) to obtain useful information from the images.

[0047] (2) Candidate region generation: Generate candidate regions that may contain target objects through, for example, selective search.

[0048] (3) Classification and positioning: Classify the candidate area, determine whether it contains the target object, and accurately locate the bounding box of the target object.

[0049] In some object detection algorithms, feature extraction and classification positioning can also be performed without generating candidate regions.

[0050] (4) Target box regression: Correct the position and size of the target box according to the predicted position offset to improve detection accuracy.

[0051] Target detection algorithms are mainly divided into two categories: two-stage target detection algorithms and one-stage target detection algorithms.

[0052] Two-stage target detection algorithm: first generate candidate regions, then classify and locate the candidate regions. Two-stage target detection algorithms include R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN, Faster R-CNN, etc. Among them, the R-CNN target detection algorithm applies deep learning technology to the field of target detection. The Fast R-CNN target detection algorithm improves R-CNN and significantly improves the speed and accuracy of target detection. Faster R-CNN introduces Region Proposal Networks (RPN), which can efficiently generate candidate regions and improve the speed and accuracy of target detection.

[0053] One-stage target detection algorithm: directly extracts features and classifies and locates the entire image without generating candidate regions. Examples of one-stage target detection algorithms include YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). The YOLO target detection algorithm treats target detection as a regression problem and can process images in real time at a relatively high speed. The SSD target detection algorithm is a target detection algorithm based on a deep convolutional neural network that achieves a good balance between speed and accuracy.

[0054] Considering the huge amount of data in actual field use, in order to reduce the processing pressure of the system, the first filtering will be performed according to the detection results. Perform target detection on each frame of the surveillance video, and output the detection results of the human body and the bag with the bag strap mark detected in one frame. That is, if the human body and the bag with the bag strap mark are detected in one frame, the detection frame information of these two types of targets will be output. Each detection frame contains the detection frame coordinates and the detection frame confidence, which can be represented by x1, y1, x2, y2, score, where x1, y1, x2, y2 represent the coordinates of the two vertices on the diagonal of the detection frame, and score represents the confidence of the detection frame; if the human body and the bag with the bag strap mark are not detected in one frame, no subsequent processing will be performed.

[0055] Next, the human body frame coordinates and scores, luggage frame coordinates and scores output by the target detection algorithm are input into the target tracking algorithm for prediction and matching. The tracking results of each human body and each luggage are obtained through the target tracking algorithm, including position, tracking confidence and tracking id. Each tracking result can be expressed as x1, y1, x2, y2, conf, id, where x1, y1, x2, y2 represent the coordinates of the two vertices on the diagonal of the target frame, conf represents the tracking confidence, and id is the serial number of each target (tracking id). Further, pid can be used to represent the tracking id of the human body, and bid can be used to represent the tracking id of the luggage.

[0056] In subsequent frames, the target tracking algorithm finds the area most similar to the target through feature matching, and updates the target's position and status information based on the matching results. Target tracking algorithms include traditional tracking algorithms such as Mean-Shift, CamShift, and KCF that track by iteratively updating the target position, as well as deep learning tracking algorithms such as SiameseFC and DeepSORT that use convolutional neural networks to extract features and track.

[0057] To reduce the processing pressure of the system, filtering will be performed again based on the tracking results. Images in which there is an intersection between the detected human body and the luggage with a strap logo in the detection results will be tracked, and other images will not be tracked. That is, the human body and the luggage with a strap logo in a frame of image are matched to form a pair of human body and luggage (human-luggage pair), which can be marked as pbpairs_ij<pid, bid>, where 0 < i ≤ person_nums, 0 < j ≤ bags_nums, persons_num represents the number of human bodies output by the object detection algorithm, and bags_num represents the number of luggage with a strap logo output by the object detection algorithm. Thus, a sequence of human-luggage pairs is obtained based on multiple frames of images. For each pair of human body and luggage, calculate the iou (intersection over union) of the human body detection box and the luggage detection box. If iou > 0, it means there is an intersection between the human body detection box and the luggage detection box, then keep this pair of records, otherwise delete this pair of records.

[0058] According to the filtered tracking results, if a pair of human body and luggage pbpairs_ij<pid, bid> is continuously established for more than T frames, then starting from the time when the pair of human body and luggage is established, extract t frames from the video image sequence at intervals as key frame images, 0 < t < T. The t key frame images and the detection results of the corresponding valid pairs of human body and luggage are used as the input of the subsequent spatio-temporal interaction behavior recognition network for the human body to the luggage to detect whether there is an abnormal behavior of the human body breaking the luggage.

[0059] Based on the multiple frames of images obtained by detection and tracking with the same pair of human body and luggage, combined with the detection box information of the human body and the detection box information of the luggage in each frame of image, determine the sequence of images of the pair of human body and luggage and the sequence of coordinates of the pair of human body and luggage. Combined with the detection box information of the human body in each frame of image, determine the sequence of human body images. In the sequence, the image of the pair of human body and luggage is, for example, the image within the (minimum) circumscribed rectangle of the pair of human body and luggage, the coordinates of the pair of human body and luggage include, for example, the human body detection box, the luggage detection box, and the intersection detection box of the human body and the luggage within the (minimum) circumscribed rectangle of the pair of human body and luggage, and the human body image is, for example, the image within the human body detection box. Thus, an effective sequence is obtained to improve the behavior recognition effect.

[0060] Figure 2 A schematic diagram showing some embodiments of the present disclosure using a spatio-temporal interaction behavior recognition network for the human body to the luggage to detect whether there is an abnormal behavior of breaking the strap logo of the luggage. For details, refer to steps 120 - 150.

[0061] In step 120, process the sequence of images of the pair of human body and luggage to determine the first feature representing the spatio-temporal interaction characteristics of the human body to the luggage.

[0062] For example, a feature extraction network is used to extract the visual features of each human body and luggage pair image in the human body and luggage pair image sequence; the visual features of each human body and luggage pair image are averaged and pooled to obtain the first feature. Thus, the overall features or trends of the entire sequence regarding the spatiotemporal interaction between human body and luggage are obtained.

[0063] Assume that the visual features of the i-th frame of the human body and luggage pair image sequence are , 𝑖={1,…,t}, the visual features of the first frame of the human body and luggage image To the visual features of the human body and luggage pair images in the tth frame Average pooling process to obtain the first feature , the formula is:

[0064] .

[0065] The feature extraction network uses deep learning technology to extract high-level abstract features from the image through multi-layer convolution, pooling and other operations. These features can more effectively represent the image content, thereby improving the performance of subsequent processing tasks.

[0066] Feature extraction networks include, for example, VGGNet, ResNet (residual network), InceptionNet (also known as GoogleNet), MobileNet, DenseNet (densely connected network), CSPDarknet53 in YOLOv5, etc., but are not limited to the examples given. Among them, VGGNet is a deep convolutional neural network, which is widely used in computer vision tasks, including target detection, image classification, etc. Its network structure is relatively simple, and it mainly builds a deep model by stacking multiple 3x3 convolution kernels and 2x2 pooling layers. ResNet effectively alleviates the gradient vanishing problem in deep neural networks by introducing residual connections, making it easier to train deeper network structures, and performs well in multiple computer vision tasks, such as image classification, target detection, etc. InceptionNet is a multi-convolution kernel network that uses convolution kernels of different sizes to extract features in parallel, which improves the network's receptive field and feature richness. It combines convolution operations of multiple scales and can capture image features at different scales. At the same time, the introduction of 1x1 convolution kernels effectively reduces computational complexity. MobileNet is a lightweight deep neural network designed to achieve real-time target detection on mobile devices. It uses depthwise separable convolution to reduce model parameters and computation, so that the model has low computational complexity and fast reasoning speed while maintaining high prediction accuracy. DenseNet improves feature reuse and network expression ability through dense connection. The output of each convolutional layer is connected to the input of all convolutional layers after this layer. This dense connection method helps to reduce the gradient disappearance problem and enhance the feature expression ability of the network. CSPDarknet53 in YOLOv5 is a convolutional neural network structure that uses cross-stage partial (CSP) to reduce the dimension of feature maps, thereby reducing the number of model parameters. CSP divides the input feature map into two parts for processing, and finally connects the two parts of the feature map as the input of the next layer. This structure enables YOLOv5 to have a faster reasoning speed while maintaining high detection accuracy.

[0067] In step 130, the human body image sequence is processed to determine a second feature that characterizes the temporal and spatial posture feature of the human body. This step is implemented, for example, by the following steps.

[0068] (1) Detect each human body key point in each human body image in the human body image sequence, and connect each human body key point to obtain a human body posture graph.

[0069] The human body key points, for example, include 17 human body parts corresponding to the human skeleton key points, including: 0, nose, 1, left eye, 2, right eye, 3, left ear, 4, right ear, 5, left shoulder, 6, right shoulder, 7, left elbow, 8, right elbow, 9, left wrist, 10, right wrist, 11, left hip, 12, right hip, 13, left knee, 14, right knee, 15, left ankle, 16, right ankle.

[0070] Examples of human key point detection networks include CPM (Convolutional Pose Machines): a pose machine is constructed through a convolutional neural network; the key point positions are gradually refined; Hourglass: an hourglass-shaped network structure is used to improve detection accuracy through multi-scale feature fusion; OpenPose: multi-person key point detection is performed in combination with Part Affinity Fields, which can simultaneously detect the key points of multiple people in the image and correctly distinguish the ownership of each key point; CPN (Cascaded Pyramid Network): through a cascaded pyramid network structure, the difficult problems of key point detection, such as occlusion and complex postures, are gradually solved; HRNet (Deep High-Resolution Representation Learning for Human Pose Estimation): by maintaining the high-resolution representation of the network, feature extraction and fusion are performed simultaneously at multiple scales to improve detection accuracy.

[0071] (2) Based on the human posture graph of each human body image, each human body image is encoded to obtain a human posture feature graph.

[0072] An exemplary encoding method is: encoding the pixel values ​​on the lines connecting different parts of the human body posture diagram of each human body picture as different values ​​in the first set of values; encoding the pixel values ​​outside the lines connecting the parts of the human body posture diagram of each human body picture as a second value. This embodiment is not limited to this encoding method, and any encoding method that can distinguish the human body posture information from other information in the human body picture is acceptable.

[0073] Taking the i-th frame of human body picture as an example, 𝑖={1,…,t}, connect each human key point to get the human body posture graph of the i-th frame of human body picture , Represents the characteristic domain of the two-dimensional coordinates of 17 key points of the human body; using the function Indicates encoding of key points of the human body: , where each line connecting the key points of the human body is encoded as a different value 𝑥∈(0,1) to represent different parts of the body. This helps the model recognize and distinguish different human postures. The remaining pixel values ​​outside the line are encoded as 0, thereby completing the encoding of the i-th frame of the human body image.

[0074] In addition, after encoding each human body image, it can also be adjusted to a fixed scale to obtain a human body posture feature map. In this way, a human body posture feature map with consistent size is obtained, which is convenient for subsequent processing.

[0075] (3) Perform convolution and spatial maximum pooling on each human posture feature map to extract the human body spatial posture features of each human posture feature map, thereby capturing the posture features of the human body in the spatial dimension.

[0076] Each human posture feature map can be downsampled and then subjected to convolution processing and spatial maximum pooling processing to improve processing efficiency.

[0077] (4) Perform time average pooling processing on each human body spatial posture feature to obtain the second feature. This continues to capture the key information of the time series and ultimately obtains the posture features of the human body in the spatiotemporal dimension.

[0078] In step 140, the coordinate sequences of the human body and the luggage are processed to determine the third feature that characterizes the spatiotemporal mask features of the human body and the luggage. This step is implemented, for example, by the following steps.

[0079] (1) In the bounding rectangle of each human body and luggage pair in the human body and luggage pair coordinate sequence, generate spatial masks of the human body, luggage, and the intersection of the human body and luggage based on the human body detection frame, luggage detection frame, and the intersection detection frame of the human body and luggage.

[0080] Therefore, for the i-th coordinate sequence of the human body and luggage pair, 𝑖={1,…,t}, three spatial masks are generated in the circumscribed rectangular box of the human body and luggage pair: , corresponding to the human body area, luggage area, and human body and luggage intersection area respectively.

[0081] (2) Based on the spatial mask in the bounding rectangle of each human body and luggage pair, the bounding rectangle of each human body and luggage pair is encoded to obtain the spatial mask encoding.

[0082] An exemplary encoding method includes: encoding the value within the spatial mask in the circumscribed rectangular frame of each human body and luggage pair as a third value, such as a non-zero value such as 1; encoding the value outside the spatial mask in the circumscribed rectangular frame of each human body and luggage pair as a fourth value.

[0083] In addition, the circumscribed rectangular frame of the human body and the bag pair can be adjusted to a fixed scale in order to obtain a spatial mask encoding of a consistent size, which is convenient for subsequent processing.

[0084] (3) Each spatial mask encoding is subjected to convolution and spatial maximum pooling to extract spatial mask features.

[0085] (4) Performing temporal average pooling processing on each spatial mask feature to obtain the third feature, which is used to characterize the spatiotemporal mask features of the human body and luggage.

[0086] In step 150, the first feature, the second feature and the third feature are fused and classified to obtain a recognition result of a human body's breaking behavior of a bag strap label.

[0087] An exemplary fusion and classification method includes: And the second feature After splicing, feature fusion is performed through a fully connected network, and the fused features are input into a classification function, such as a sigmoid function, to obtain the first classification result. ; The third feature The second classification result is obtained by processing through the fully connected network and classification function (such as sigmoid function) in turn. ; Fusion of the first classification results And the second classification results , and obtain the recognition result of the human body's breaking behavior of the bag strap label An exemplary fusion method is: In addition, the first classification result can also be And the second classification results Set different fusion weights for fusion.

[0088] On the basis of utilizing the changes in human posture characteristics, we pay attention to the characteristics of the intersection area between people and luggage, and at the same time incorporate information from time and space dimensions. By integrating various features, the accuracy of breaking and unlocking behavior recognition is increased.

[0089] The behavior recognition model for implementing the behavior recognition method includes: an algorithm for detecting and tracking targets such as human bodies and bags with bag strap labels, and a spatiotemporal interactive behavior recognition network for detecting whether a human body has an abnormal behavior of breaking the bag strap labels.

[0090] The behavior recognition model for implementing the behavior recognition method is used for behavior recognition after training. In the process of training the behavior recognition model for implementing the behavior recognition method, the parameters of the behavior recognition model are updated according to the difference information between the predicted value of the recognition result of the demolition behavior output by the behavior recognition model and the reference value of the recognition result of the demolition behavior. The reference value of the recognition result of the demolition behavior is known information and can be generated by annotation.

[0091] During the training process of the behavior recognition model, the algorithms used to detect and track targets such as human bodies and bags with strap labels, and the spatiotemporal interactive behavior recognition network used to detect abnormal behavior of human bodies breaking the strap labels of bags, can be trained together; or the target detection and tracking algorithm can be implemented using an existing algorithm, and the spatiotemporal interactive behavior recognition network can be trained separately.

[0092] Figure 3 A schematic diagram showing a behavior recognition device according to some embodiments of the present disclosure.

[0093] like Figure 3 As shown, the behavior recognition device 300 includes: units 310-320.

[0094] The detection and tracking unit 310 is configured to detect and track a human body and a bag with a bag strap mark in a surveillance video to obtain a human body and bag pair image sequence, a human body image sequence, and a human body and bag pair coordinate sequence.

[0095] The detection and tracking unit 310 is configured to perform target detection on each frame of the surveillance video, and output a detection result in which a human body and a bag with a bag strap mark are detected in one frame of the image.

[0096] The detection and tracking unit 310 is configured to track the image in which the human body and the bag with the bag strap mark intersect in the detection result.

[0097] The detection and tracking unit 310 or the behavior recognition unit 320 is configured to determine a human body and luggage pair image sequence, a human body and luggage pair coordinate sequence based on multiple frames of images with the same human body and luggage pair obtained through detection and tracking, combined with the detection frame information of the human body and the detection frame information of the luggage in each frame, and determine a human body image sequence combined with the detection frame information of the human body in each frame.

[0098] The behavior recognition unit 320 is configured to process a sequence of images of human bodies and luggage pairs to determine a first feature that characterizes the spatiotemporal interaction characteristics of human bodies with luggage; process a sequence of images of human bodies to determine a second feature that characterizes the spatiotemporal posture characteristics of human bodies; process a sequence of coordinates of human bodies and luggage pairs to determine a third feature that characterizes the spatiotemporal mask characteristics of human bodies and luggage; and fuse and classify the first feature, the second feature, and the third feature to obtain an identification result of the human body's behavior of breaking the strap label of luggage.

[0099] The behavior recognition unit 320 includes a first feature processing module 321, which is configured to extract visual features of each human body and luggage pair image in the human body and luggage pair image sequence; perform average pooling processing on the visual features of each human body and luggage pair image to obtain the first feature.

[0100] The behavior recognition unit 320 includes a second feature processing module 322, which is configured to detect each human key point of each human body picture in the human body picture sequence, and connect each human key point to obtain a human body posture graph; based on the human body posture graph of each human body picture, encode each human body picture to obtain a human body posture feature graph; perform convolution processing and spatial maximum pooling processing on each human body posture feature graph to extract human body spatial posture features; perform time average pooling processing on each human body spatial posture feature to obtain the second feature. Wherein, based on the human body posture graph of each human body picture, encoding each human body picture includes: encoding the pixel values ​​on the connecting lines of different parts in the human body posture graph of each human body picture as different values ​​in the first group of values; encoding the pixel values ​​outside the connecting lines of each part in the human body posture graph of each human body picture as the second value.

[0101] The behavior recognition unit 320 includes a third feature processing module 323, which is configured to generate a spatial mask of a human body, a luggage, and an intersection of a human body and a luggage in the bounding rectangle of each human body and luggage pair in the coordinate sequence of the human body and luggage pair; based on the spatial mask in the bounding rectangle of each human body and luggage pair, encode the bounding rectangle of each human body and luggage pair to obtain a spatial mask code; perform convolution processing and spatial maximum pooling processing on each spatial mask code to extract spatial mask features; perform time average pooling processing on each spatial mask feature to obtain the third feature. Wherein, encoding the bounding rectangle of each human body and luggage pair includes: encoding the value in the spatial mask in the bounding rectangle of each human body and luggage pair as a third value; encoding the value outside the spatial mask in the bounding rectangle of each human body and luggage pair as a fourth value.

[0102] The behavior recognition unit 320 includes a behavior recognition module 324, which is configured to concatenate the first feature and the second feature, perform feature fusion through a fully connected network, input the fused feature into a classification function, and obtain a first classification result; process the third feature through the fully connected network and the classification function in sequence to obtain a second classification result; and fuse the first classification result and the second classification result to obtain an identification result of a human body's behavior of breaking the strap label of a bag.

[0103] Figure 4 A schematic diagram showing a behavior recognition device according to some embodiments of the present disclosure.

[0104] like Figure 4 As shown, the behavior recognition device 400 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410 , and the processor 420 is configured to execute the behavior recognition method in each embodiment based on the instructions stored in the memory 410 .

[0105] The behavior recognition device 400 may further include an input / output interface 430 , a network interface 440 , a storage interface 450 , etc. These interfaces 430 , 440 , 450 , the memory 410 , and the processor 420 may be connected, for example, via a bus 460 .

[0106] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0107] The processor 420 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors or other discrete hardware components.

[0108] Among them, the input and output interface 430 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networked devices. The storage interface 450 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 460 can use any bus structure among a variety of bus structures. For example, the bus structure includes but is not limited to the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0109] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0110] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more (non-transient) computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that implements its solution mainly through a computer program.

[0111] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A behavior recognition method, characterized in that: include: Detect and track human bodies and bags with bag strap labels in surveillance videos to obtain image sequences of human bodies and bags, human body image sequences, and human bodies and bags coordinate sequences; Processing a sequence of images of human bodies and luggage pairs, and performing average pooling processing on visual features of each image of human bodies and luggage pairs in the sequence of images of human bodies and luggage pairs, so as to determine a first feature representing a spatiotemporal interaction feature of human bodies and luggage pairs; Processing the human body picture sequence, performing temporal average pooling processing on the human body spatial posture features corresponding to each human body picture in the human body picture sequence, so as to determine a second feature representing the temporal and spatial posture features of the human body; Processing the coordinate sequence of human body and luggage pair, performing temporal average pooling processing on the spatial mask features corresponding to each human body and luggage pair coordinate in the coordinate sequence of human body and luggage pair, so as to determine the third feature representing the spatiotemporal mask features of human body and luggage; The first feature, the second feature, and the third feature are fused and classified to obtain a recognition result of a human body's breaking behavior of a bag strap label.

2. The behavior recognition method according to claim 1, characterized in that: Detection of human bodies and bags with bag straps in surveillance videos includes: Perform target detection on each frame of the surveillance video and output the detection results of both the human body and the bag with the bag strap mark detected in one frame.

3. The behavior recognition method according to claim 2, characterized in that: Tracking of people and bags with bag straps in surveillance video includes: Track the images in which the human body and the bag with the bag strap mark intersect in the detection results.

4. The behavior recognition method according to claim 3, characterized in that: Obtaining a human body and luggage pair image sequence, a human body image sequence, and a human body and luggage pair coordinate sequence includes: Based on multiple frames of images with the same human body and luggage pair obtained by detection and tracking, the human body detection frame information and the luggage detection frame information in each frame are combined to determine the human body and luggage pair image sequence and the human body and luggage pair coordinate sequence; the human body image sequence is determined by combining the human body detection frame information in each frame.

5. The behavior recognition method according to claim 1, characterized in that: Processing of human body and luggage image sequences includes: Extract visual features of each human body and luggage pair image in the human body and luggage pair image sequence; Average pooling is performed on the visual features of each human body and luggage pair image to obtain the first feature.

6. The behavior recognition method according to claim 1, characterized in that: Processing of human body image sequences includes: Detect each human key point of each human body picture in the human body picture sequence, and connect each human key point to obtain a human body posture graph; Based on the human posture graph of each human body picture, each human body picture is encoded to obtain a human posture feature graph; Perform convolution processing and spatial maximum pooling processing on each human posture feature map to extract the human body spatial posture features; Perform time average pooling processing on each human body spatial posture feature to obtain the second feature.

7. The behavior recognition method according to claim 6, characterized in that: Based on the human posture graph of each human body picture, encoding each human body picture includes: Encoding pixel values ​​on the lines connecting different parts of the human body posture diagram of each human body image as different values ​​in the first set of values; The pixel values ​​outside the connecting lines of each part in the human body posture diagram of each human body picture are encoded as a second value.

8. The behavior recognition method according to claim 1, characterized in that: Processing of coordinate sequences of human body and luggage includes: Generate a spatial mask of the human body, the luggage, and the intersection of the human body and the luggage in the bounding rectangle of each human body and luggage pair in the human body and luggage pair coordinate sequence; Based on the spatial mask in the circumscribed rectangular frame of each human body and luggage pair, encode the circumscribed rectangular frame of each human body and luggage pair to obtain the spatial mask encoding; Each spatial mask encoding is subjected to convolution and spatial maximum pooling to extract spatial mask features; Temporal average pooling is performed on each spatial mask feature to obtain the third feature.

9. The behavior recognition method according to claim 8, characterized in that: Encoding the bounding rectangle of each human body and luggage pair includes: Encode the value in the spatial mask of the bounding rectangle of each human body and luggage pair as a third value; The values ​​outside the spatial mask in the bounding rectangle of each human body and luggage pair are encoded as the fourth value.

10. The behavior recognition method according to claim 1, characterized in that: The first feature, the second feature, and the third feature are integrated and classified to obtain the recognition results of the human body's breaking behavior of the bag strap label, including: After concatenating the first feature and the second feature, the features are fused through a fully connected network, and the fused features are input into the classification function to obtain a first classification result; The third feature is processed by the fully connected network and the classification function in sequence to obtain the second classification result; The first classification result and the second classification result are integrated to obtain the recognition result of the human body's breaking behavior of the bag strap label.

11. The behavior recognition method according to claim 1, characterized in that: Also includes: In the process of training the behavior recognition model for implementing the behavior recognition method, the parameters of the behavior recognition model are updated according to the gap information between the predicted value and the reference value of the recognition result of the demolition behavior output by the behavior recognition model.

12. A behavior recognition device, comprising: Memory; And a processor coupled to the memory, characterized in that the processor is configured to execute the behavior recognition method according to any one of claims 1-11 based on instructions stored in the memory.

13. A behavior recognition device, characterized in that: include: A detection and tracking unit is configured to detect and track a human body and a bag with a bag strap mark in a surveillance video to obtain a human body and bag pair picture sequence, a human body picture sequence, and a human body and bag pair coordinate sequence; The behavior recognition unit is configured to process the sequence of images of human body and luggage pairs, and determine a first feature representing the spatiotemporal interaction feature of the human body and luggage pair images by performing average pooling processing on the visual features of each human body and luggage pair image in the sequence of human body and luggage pair images; Processing the human body picture sequence, performing temporal average pooling processing on the human body spatial posture features corresponding to each human body picture in the human body picture sequence, so as to determine a second feature representing the temporal and spatial posture features of the human body; Processing the coordinate sequence of human body and luggage pair, performing temporal average pooling processing on the spatial mask features corresponding to each human body and luggage pair coordinate in the coordinate sequence of human body and luggage pair, so as to determine the third feature representing the spatiotemporal mask features of human body and luggage; The first feature, the second feature, and the third feature are fused and classified to obtain a recognition result of a human body's breaking behavior of a bag strap label.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the behavior recognition method described in any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the behavior recognition method described in any one of claims 1 to 11 are implemented.

Images