A method and system for identifying interactive behaviors of people in post office scenes

Abstract

The present invention discloses a method and system for identifying interactive behaviors of people in post office scenes, belonging to the field of video analysis technology. The method utilizes background subtraction to determine if abnormal behavior is triggered in the unpacking area, then captures a video stream. The captured video stream image is input into a pre-trained UNet network model to extract the box and human hands in the foreground. Feature descriptors for the box and human hands are constructed by combining depth and visible light information to determine the interactive relationships between people. Finally, a pre-trained SVM classifier is used to generate a judgment result, achieving accurate recognition of interactive behaviors in post office scenes. The method has a strong inhibitory effect on complex background interference, a high recognition accuracy rate for unpacking and visual inspection behaviors, and excellent robustness. It can meet the supervision requirements of express delivery stations for staff, and contributes to improving the automation and modernization level of intelligent video surveillance systems.

Description

Technical Field

[0001] The present invention relates to a method and system for identifying interactive behavior of people in a post office scene, and belongs to the technical field of video analysis. Background Art

[0002] The safety of express delivery during collection and transportation is a growing concern. To prevent the presence of flammable and explosive dangerous goods in packages, postal workers must perform an unpacking inspection before packaging. Compared to traditional manual video analysis methods, intelligent video surveillance technology can monitor and alert courier workers in real time, significantly improving monitoring efficiency and effectiveness. Therefore, it is necessary to research computer vision-based methods for identifying interactive human behavior in postal scenarios.

[0003] Current research on behavior recognition, both domestically and internationally, focuses primarily on single-person and multi-person behaviors. Depending on the type of data used, mainstream interactive behavior recognition and analysis methods can be categorized as those based on visible light (RGB) information and those based on depth information. Early research typically employed RGB-based methods, such as manually constructing oriented gradient histogram features and extracting motion features using optical flow. While traditional RGB-based interactive behavior recognition methods offer significant advantages in certain scenarios, they still face numerous challenges. For example, they are susceptible to interference from background and lighting, making them difficult to achieve in all-weather video surveillance applications.

[0004] With the emergence and widespread use of depth cameras, interactive behavior recognition methods based on depth information have begun to gain traction. Compared to methods that rely solely on RGB information, depth-based interactive behavior recognition methods introduce a new dimension of information. Leveraging the collected three-dimensional joint information allows for better analysis of human behavior while also avoiding the traditional RGB method's reliance on factors such as color and brightness.

[0005] Pisharady PK, et al. used the 3D joint point data collected by the depth camera to convert it into joint angle features and used the support vector machine (SVM) to identify human behavior (Pisharady PK, Saerbeck M. Kinect based body posture detection and recognition system [C] / / International Conference on Graphic and Image Processing (ICGIP 2012). International Society for Optics and Photonics, 2013, 8768: 87687F.); Yang Wenlu et al. combined the variance of the angle change of the skeleton points and the spatial relationship between the skeleton points to extract the key posture, and realized the accurate recognition of two-person interaction behavior (Yang Wenlu, Yu Mengmeng, Xie Hong. Two-person interaction behavior recognition based on key posture [J]. Computer Applications, 2020, 40(08): 2231-2235.); Cheng Keyang et al. proposed a skeleton-based spatiotemporal modeling method, which realized the effective representation of the interaction skeleton graph (Cheng Keyang, Wu Jinxia, ​​Wang Wenshan, et al. Multi-person interaction behavior recognition integrated with spatiotemporal graph convolution [J]. Journal of Image and Graphics, 2021, 26(07): 1681-1691.).

[0006] However, during the unboxing and inspection process, if only human skeleton information is used, the semantic information of the box will be lost. If the characteristics of the "object" in the human-object interaction process cannot be identified, the interference of some similar behaviors cannot be eliminated, thereby affecting the recognition accuracy of the unboxing and inspection interaction behavior. As for how to accurately describe the specific behavioral patterns of people and objects in the post office scenario, the above research has not provided a good reference and solution. Summary of the Invention

[0007] In order to solve the problem of poor accuracy in human-object interaction behavior recognition in the current interactive recognition solution in the post office unpacking and inspection scenario, the present invention provides a method and system for identifying human interaction behavior in post office scenarios. The technical solution is as follows:

[0008] A first object of the present invention is to provide a method for identifying interactive behaviors of people in a post office scene, the method comprising:

[0009] Step 1: Use background subtraction to determine whether there is any behavior in the detection area. If there is behavior in the detection area, the camera collects the behavior video stream, which includes: color image sequence and depth image sequence;

[0010] Step 2: Perform foreground segmentation based on the UNet network, input the color image sequence into the pre-trained UNet network model, and output the segmentation results with labels to obtain binary images of the box and the human hand;

[0011] Step 3: Register the depth image and color image;

[0012] Step 4: Using the box binary image obtained in step 2, extract features of the foreground box in the registered depth image to obtain box depth features;

[0013] Step 5: Extracting the hand motion trajectory based on the binary hand image obtained in step 2 to obtain the hand motion features;

[0014] Step 6: Based on the box depth features and hand motion features, construct a feature descriptor containing depth and spatiotemporal information;

[0015] Step 7: Use the pre-trained SVM classifier to classify the feature descriptors to achieve recognition and detection of human interaction behaviors.

[0016] Optionally, the fourth step uses the box foreground depth value variance of a single-frame image during the unpacking process as the feature value representing the image. The depth variance is calculated as follows:

[0017]

[0018] Among them, σ 2 represents the variance of the depth value of the foreground image, n represents the total number of pixels in the foreground image, d k Indicates the depth value of the k-th pixel, d m Represents the average depth value of all pixels in the foreground image;

[0019] By calculating the variance σ of the depth foreground map of the single frame image box 2 , and then multi-frame image variance σ is converted into 2 Forming the eigenvector λ bn .

[0020] Optionally, the process of extracting hand motion features in step 5 includes:

[0021] Extract feature points from the binary image of the hand. Select the hand farthest from the camera and perform pixel traversal on its binary image. Use the row number of the top white pixel in the image as the ordinate and the column number of the leftmost white pixel in the image as the abscissa to obtain the coordinates of the feature point F(x,y).

[0022] Calculate the pixel distance D from the feature point F(x,y) to the image coordinate origin t, the distance D between the feature points of the continuous frame images and the origin is calculated according to the time sequence t Forming the eigenvector λ hn .

[0023] Optionally, the process of using the background subtraction method in step 1 to determine whether an action occurs in the detection area includes:

[0024] Compare the input image with the background model, and extract and segment the moving target in the image through changes in grayscale, chroma and other features. The calculation formula of the background subtraction method is:

[0025] d (k) (x,y)=|f (k+1) (x,y)-B (k) (x,y)|

[0026]

[0027] Among them, d (k) (x,y) represents the current frame image f( k+1) (x,y) and background image B (k) The difference between (x,y), m (k) (x, y) represents the binary image of the difference image, T represents the threshold of the pixel difference between the current frame image and the background image, and when it is greater than or equal to T, it is judged that there is a foreground target;

[0028] When there is no moving target entering the camera's field of view, the detection system is in standby mode and only acquires video streams and image frames. The first frame image f0(x, y) is selected as the initial background image B0(x, y). If the scene changes, the background image is updated accordingly.

[0029] When a moving target appears in the detection area, the binary image m (k) There are a lot of white pixels in (x, y). We calculate the ratio s of white pixels to the total pixels of the image and set the upper limit V of the evaluation standard based on experience. max and lower limit V min ;

[0030] When s>V min When , it is determined that there is behavior in the unpacking area, and the detection system enters the unpacking detection mode; the previous frame of the current frame is recorded as the starting point Sτ, and Sτ is used as the new background image to replace the initial background image to continue background subtraction until s <V min The behavior is determined to be finished and the current frame is recorded as the end point E τ , S τ to E τ This video frame is treated as a suspected unboxing behavior and subsequently processed;

[0031] If s is always less than V min or greater than V max , it is judged that there is a sudden change in illumination in the current frame or no behavior occurs in the detection area, and the judgment of the next frame is continued.

[0032] Optionally, the process of registering the depth image and the color image in step 3 includes:

[0033] First, restore the pixel points of the depth map to the depth coordinate system, and directly obtain the Euclidean transformation matrix T from the depth coordinate system to the color coordinate system d2c , and then map the depth point in the color coordinate system to the color plane of Z = 1. The conversion formula is:

[0034]

[0035] Among them, T d2c Represents the transformation matrix from the depth coordinate system to the color coordinate system, T w2c Represents the transformation matrix from the world coordinate system to the color coordinate system, T w2d Represents the transformation matrix from the world coordinate system to the depth coordinate system.

[0036] Optionally, during the feature extraction process, the UNet network gradually downsamples the feature map through 3×3 convolution, ReLU activation, and 2×2 maximum pooling, gradually reducing the input samples while increasing the number of channels to compensate for the loss of resolution.

[0037] Optionally, during the upsampling process, the UNet network performs 2×2 transposed convolution and 1×1 convolution on the feature map. After each upsampling, it is fused at the same scale as the number of channels corresponding to the feature extraction part, gradually restoring the image details, so that the size of the feature map gradually increases and the number of channels gradually decreases, and finally outputs a semantic segmentation image.

[0038] A second object of the present invention is to provide a post office scene character interaction behavior detection system, which is used to implement the post office scene character interaction behavior recognition method described in the claims, comprising:

[0039] Video stream acquisition equipment, used to collect behavior video streams within the detection area;

[0040] A behavior trigger module, which uses background subtraction to determine whether there is any behavior occurring in the detection area, and if so, triggers the video stream acquisition device to acquire the video stream;

[0041] The UNet foreground segmentation module is used to perform semantic segmentation on the images in the captured video stream and extract the box and human hand images in the foreground moving objects;

[0042] The feature extraction module extracts features from the box and hand images after foreground segmentation to obtain box depth features and hand motion features;

[0043] The classification output module uses the SVM classifier to identify and classify the extracted box depth features and hand motion features, and outputs the detection results.

[0044] Optionally, the video stream acquisition device includes: an RGBD camera.

[0045] The beneficial effects of the present invention are:

[0046] The present invention's method for identifying interactive behaviors of people in post office scenarios takes interactive behaviors of people in express delivery inspections at post offices as its application background. By introducing a semantic segmentation mechanism based on deep learning, a method for identifying interactive behaviors of people based on depth maps and spatiotemporal features is proposed. The method integrates the box depth features and hand motion features in the unpacking inspection process to construct a feature descriptor. Compared with existing behavior recognition methods based only on human skeleton information, the present invention effectively solves the two pain points of complex background interference in the unpacking inspection process and the difficulty in accurately describing the specific behavior patterns of people and objects.

[0047] Experiments show that using the UNet model to extract foreground objects has a strong inhibitory effect on complex background interference. Constructing feature descriptors based on RGBD multi-source information accurately describes box morphological changes and human hand behavior patterns. The proposed method for identifying interactive behaviors in postal scenarios has high accuracy for unpacking and visual inspection, exhibits excellent robustness, and can meet the supervision requirements of express delivery stations for staff, contributing to the automation and modernization of intelligent video surveillance systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a flow chart of the method for identifying human interaction behavior based on depth map and spatiotemporal features of the present invention.

[0050] Figure 2 This is the UNet network structure diagram of Example 3 of the present invention.

[0051] Figure 3 This is a comparison chart of the UNet segmentation effect and the HSV segmentation effect of Example 3 of the present invention.

[0052] Figure 4 3 is a depth map registration effect diagram of the third embodiment of the present invention, wherein (a) is a color image, (b) is a depth map before registration, and (c) is a depth map after registration.

[0053] Figure 5 This is a diagram of an unpacking behavior image sequence (box body) according to the third embodiment of the present invention.

[0054] Figure 6 3 is a depth value variance characteristic vector curve diagram of the third embodiment of the present invention.

[0055] Figure 7 This is an image sequence (human hands) of the unpacking behavior according to the third embodiment of the present invention.

[0056] Figure 8 This is a hand motion trajectory curve diagram based on feature point distances according to the third embodiment of the present invention.

[0057] Figure 9 This is a schematic diagram of the courier-object interaction behavior dataset of the present invention.

[0058] Figure 10 This is the result of the post office real scene test set, where (a) is the image sequence of unpacking behavior; (b) is the image sequence of non-unpacking behavior; and (c) is the image sequence of interference behavior.

[0059] Figure 11 It is a comparison diagram of characteristic curves, where (a) is the box depth variance curve, (b) is the box height curve, (c) is the human hand motion trajectory characteristic curve, and (d) is the human hand depth variance curve.

[0060] Figure 12 These are the results of the box type interference experiment, where (a) is the original image, (b) is the segmentation result image, (c) is the box color image, and (d) is the human hand color image.

[0061] Figure 13 These are the results of the background clutter interference experiment, where (a) is the original image, (b) is the segmentation result image, (c) is the box color image, and (d) is the human hand color image.

[0062] Figure 14 These are the results of the light change interference experiment, where (a) is the original image, (b) is the segmentation result image, (c) is the box color image, and (d) is the human hand color image.

[0063] Figure 15 Is a confusion matrix diagram, where (a) is the confusion matrix diagram under the interference of box type, (b) is the confusion matrix diagram under the interference of background clutter, and (c) is the confusion matrix diagram under the interference of light change. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0065] Example 1:

[0066] This embodiment provides a method for identifying interactive behaviors of people in a post office scene, the method comprising:

[0067] Step 1: Use background subtraction to determine whether there is any behavior in the detection area. If there is behavior in the detection area, the camera will collect the behavior video stream, which includes: color image sequence and depth image sequence;

[0068] Step 2: Perform foreground segmentation based on the UNet network. Input the color image sequence into the pre-trained UNet network model and output the segmentation results with labels to obtain binary images of the box and the human hand.

[0069] Step 3: Register the depth image and color image;

[0070] Step 4: Using the box binary image obtained in step 2, extract the features of the foreground box in the registered depth image to obtain the box depth features;

[0071] Step 5: Extract the hand motion trajectory based on the binary hand image obtained in step 2 to obtain the hand motion features;

[0072] Step 6: Based on the box depth features and hand motion features, construct a feature descriptor containing depth and spatiotemporal information;

[0073] Step 7: Use the pre-trained SVM classifier to classify the feature descriptors to achieve recognition and detection of human interaction behaviors.

[0074] Example 2:

[0075] This embodiment provides a post office scene character interaction behavior detection system, which is used to implement the post office scene character interaction behavior recognition method described in Example 1. The system includes:

[0076] Video stream acquisition equipment, used to collect behavior video streams within the detection area;

[0077] A behavior trigger module, which uses background subtraction to determine whether there is any behavior occurring in the detection area, and if so, triggers the video stream acquisition device to acquire the video stream;

[0078] The UNet foreground segmentation module is used to perform semantic segmentation on the images in the captured video stream and extract the box and human hand images in the foreground moving objects;

[0079] The feature extraction module extracts features from the box and hand images after foreground segmentation to obtain box depth features and hand motion features;

[0080] The classification output module uses the SVM classifier to identify and classify the extracted box depth features and hand motion features, and outputs the detection results.

[0081] Example 3:

[0082] This embodiment provides a method for identifying interactive behaviors of people in a post office scene.

[0083] To address the problems of poor interference suppression and loss of semantic information in traditional interactive behavior recognition methods based on RGB information, this embodiment introduces a UNet-based semantic segmentation algorithm to accurately extract boxes and human hands in complex backgrounds, obtaining the target category and position information; it also integrates depth and visible light information to accurately describe the box shape changes and human hand behavior patterns during the unpacking inspection process, and determine the human interaction relationship; finally, it trains an SVM classifier to identify and analyze human interaction behaviors, achieving high accuracy and robustness in a post office environment with severe interference from background clutter, light changes, and other factors.

[0084] The process of the human interaction behavior recognition method based on depth map and spatiotemporal features proposed in this embodiment is as follows: Figure 1 As shown in the figure, after the detection system uses background subtraction to determine that an abnormal behavior has been triggered in the unpacking area, it captures a video stream. The captured video stream image is input into a pre-trained UNet network model to extract the box and human hand in the foreground, obtaining the target category and location information. Depth and visible light information are combined to construct feature descriptors for the box and human hand, and the interaction between people is determined. Finally, a pre-trained SVM classifier is used to generate a judgment result, achieving accurate recognition of human interaction in the post office scene. By distinguishing between unpacking and non-unpacking behaviors, the standardization of express delivery operations is ensured.

[0085] The foreground object extraction method based on semantic segmentation in this embodiment specifically includes:

[0086] Step 1. Use abnormal behavior trigger detection based on background subtraction. Use background subtraction to determine whether any behavior occurs in the unpacking area.

[0087] The basic idea of ​​background subtraction is to compare the input image with the background model and to extract and segment the moving targets in the image through changes in grayscale, chroma and other features.

[0088] d (k) (x,y)=|f (k+1) (x,y)-B (k) (x,y)| (1)

[0089]

[0090] The calculation formula of background subtraction method is shown in formula (1) and formula (2), where d (k) (x,y) represents the current frame image f( k+1) (x,y) and background image B (k) The difference between (x,y), m (k) (x, y) represents the binary image of the difference image, T represents the threshold of the pixel difference between the current frame image and the background image, and when it is greater than or equal to T, it is judged that there is a foreground target;

[0091] When there is no moving target entering the camera's field of view, the detection system is in standby mode and only acquires video streams and image frames. The first frame image f0(x, y) is selected as the initial background image B0(x, y). If the scene changes, the background image will be updated accordingly.

[0092] When a moving target appears in the detection area, the binary image m (k) There are a lot of white pixels in (x, y). We calculate the ratio s of white pixels to the total pixels of the image and set the upper limit V of the evaluation standard based on experience. max and lower limit V min The extraction of moving targets can be achieved. min When , it is determined that there is an action in the unpacking area, and the detection system enters the unpacking detection mode. The previous frame of the current frame is recorded as the starting point Sτ, and Sτ is used as the new background image to replace the initial background image to continue background subtraction until s <V min The behavior is determined to be finished and the current frame is recorded as the end point E τ . τ to E τ This video frame is treated as a suspected unpacking behavior for subsequent processing. If s is always less than V min or greater than V max , it is judged that there is a sudden change in illumination in the current frame or no behavior occurs in the detection area, and the judgment of the next frame is continued.

[0093] Step 2. UNet-based foreground segmentation. After entering the unboxing detection mode, the foreground moving target needs to be extracted. After extraction, the box and human hand in the foreground moving target need to be further segmented based on semantic information. Because the traditional HSV threshold-based segmentation method cannot distinguish between human hands and boxes, which are usually very similar in color, and interference factors such as background debris and light changes also have a significant impact on the accurate extraction of features, this embodiment chooses to use a convolutional neural network (CNN) to extract the foreground box and human hands separately.

[0094] UNet is a popular deep learning model for semantic segmentation. The UNet model has a simple structure and only requires a small number of training samples to achieve better segmentation performance. It combines low-level detail information with high-level semantic information to achieve better segmentation of boxes and hands. The UNet network was first used in medical image segmentation. Its network structure consists of two parts: feature extraction and upsampling. Figure 2 As shown in the figure, the UNet network structure diagram used in this embodiment is given. During the feature extraction process, the feature map is gradually downsampled through 3×3 convolution + ReLU activation and 2×2 maximum pooling, the input samples are gradually reduced, and the number of channels is increased to compensate for the loss of resolution. During the upsampling process, the feature map is subjected to 2×2 transposed convolution and 1×1 convolution. After each upsampling, it is fused at the same scale as the number of channels corresponding to the feature extraction part, gradually restoring the image details, so that the size of the feature map gradually increases and the number of channels gradually decreases, and finally the semantic segmentation image is output.

[0095] When the detection system enters the unboxing detection mode, the system will automatically save the color image and depth image of the suspected unboxing behavior, input the color image sequence into the pre-trained UNet network model, and output the labeled segmentation results to obtain the binary image of the box and the human hand.

[0096] like Figure 3 As shown in the figure, a comparison of the UNet segmentation effect and the HSV segmentation effect is given. Compared with the traditional HSV threshold-based method, the UNet model has better segmentation results for boxes and hands of different types and colors. In the case of similar colors between the human hand and the box, the UNet model trained in this embodiment has a clear advantage over the traditional method, achieving accurate segmentation and extraction, ensuring that semantic information is not lost, and obtaining complete category and position information of the foreground object.

[0097] OpenPose is used to calculate the key frames to obtain the bone points and limb feature vectors, which are then input into the deep learning behavior classification network to obtain the initial behavior pattern and position of the human body.

[0098] The method for extracting human-object interaction features based on depth information in this embodiment specifically includes:

[0099] Step 1. Depth map registration. Due to the physical location difference between the visible light camera and the depth camera, in order to use the RGB image segmentation results to process the box depth image in the next step, the box depth map and color image need to be registered. That is, the image coordinate system of the depth map needs to be converted to the image coordinate system of the color image.

[0100] First, restore the pixel points of the depth map to the depth coordinate system, and directly obtain the Euclidean transformation matrix T from the depth coordinate system to the color coordinate systemd2c Finally, the depth point in the color coordinate system is mapped to the color plane of Z = 1. The specific calculation formula is shown in formula (3):

[0101]

[0102] Among them, T d2c Represents the transformation matrix from the depth coordinate system to the color coordinate system, T w2c Represents the transformation matrix from the world coordinate system to the color coordinate system, T w2d Represents the transformation matrix from the world coordinate system to the depth coordinate system. According to this formula, the depth map and the color map can be registered. The depth map and the color map before and after registration are as follows: Figure 4 As shown, (a) is the color image, (b) is the depth image before registration, and (c) is the depth image after registration.

[0103] Step 2. Box depth feature extraction. In human interaction behavior analysis, objects, as the subject of the behavior, have representative motion characteristics. During the unboxing inspection process, using only human skeleton information would result in a loss of semantic information about the box. Therefore, this embodiment extracts box depth features as a basis for judging unboxing behavior and determining human interaction relationships.

[0104] After obtaining the registered depth map, the box binary map obtained by the UNet network can be used to extract the foreground box in the depth image.

[0105] like Figure 5 As shown in Figure 1, a sequence of images of unpacking behavior during the express inspection process is shown, including color images, depth images, and depth images of the foreground box. During the unpacking process, the depth of the box will change significantly as it opens and closes. Therefore, the variance of the foreground depth value of the box in a single frame image during the unpacking process can be selected as the feature value representing the image. The calculation formula for the depth variance is shown in Formula (4):

[0106]

[0107] Among them, σ 2 represents the variance of the depth value of the foreground image, n represents the total number of pixels in the foreground image, d k Indicates the depth value of the k-th pixel, d m Represents the average depth value of all pixels in the foreground image. By calculating the variance σ of the depth foreground image of a single frame image box 2 , and then multi-frame image variance σ is converted into 2 Forming the eigenvector λ bn , which is used as the feature vector required for SVM classifier training.

[0108] In order to more intuitively analyze the eigenvector λ between unpacking and unpacking bn In order to understand the difference, this embodiment plots the depth variance of the image sequences of 5 groups of unpacking behaviors (dashed lines) and 5 groups of non-unpacking behaviors (solid lines) in time sequence into a line graph, as shown in FIG. Figure 6 shown.

[0109] Step 3. Extraction of human hand motion features. In the analysis of human interaction behavior, humans are the main body of behavior, and their motion features also play a representative role. Figure 7 The image sequence of unpacking behavior during the express delivery inspection process is displayed, including color images, human hand mask images, and color images of the foreground human hands extracted.

[0110] This embodiment proposes a method for describing hand features based on motion trajectories, which is used to extract feature points from the binary image of the hand obtained by segmentation of the UNet network model. According to the characteristics of the unpacking behavior, the hand farthest from the camera is selected and its binary image is traversed pixel by pixel. The coordinates of the feature point F(x,y) are obtained by taking the row number of the white pixel at the top of the image as the vertical coordinate and the column number of the white pixel at the leftmost side of the image as the horizontal coordinate. Calculate the pixel distance D from the feature point F(x,y) to the origin of the image coordinates. t , the distance D between the feature points of the continuous frame images and the origin is calculated according to the time sequence t Forming the eigenvector λ hn , which is used as the feature vector required for SVM training.

[0111] In order to more intuitively analyze the eigenvector λ between unpacking and unpacking hn In this embodiment, the feature point distance D t The calculated hand motion trajectories of the 5 groups of unpacking behaviors (dashed lines) and the 5 groups of non-unpacking behaviors (solid lines) are plotted as a line graph according to the time series, as shown in Figure 2. Figure 8 shown.

[0112] from Figure 8 It can be seen that since the hand continues to move within a certain area during the unpacking behavior, the distance between its feature points shows a "flat" trend over time, while the distance between the feature points of the non-unpacking behavior shows a "jumping" trend over time. There is a significant difference between the two. The hand feature descriptor based on motion trajectory can better reflect the changes of the hand during the human interaction behavior. Based on this, the feature vector of the hand can be constructed and the SVM classifier can be used to classify and identify it.

[0113] In this embodiment, after accurately segmenting the box and the human hand in the foreground, the box depth features λ are extracted respectively. bn and hand trajectory feature λ hn . Construct a feature vector λ containing depth and spatiotemporal information n, input the SVM classifier to classify its unpacking and unpacking behaviors, and realize the accurate recognition and detection of human interaction behaviors.

[0114] Example 4:

[0115] In order to further verify the beneficial effects of the present invention, an experiment was conducted on a method for identifying interactive behaviors of people in a post office scene as described in Example 3.

[0116] According to the recognition process designed by the present invention, the effectiveness of the human-object behavior recognition method is tested based on a self-built dataset.

[0117] The experimental environment is: Intel i7-8700 CPU, 16GB RAM, NVIDIA GTX 1660 GPU with 6GB VRAM. Operating system: Windows 10 Professional. Programming language: Python. Development platform: VSCode. Convolutional neural network framework: PyTorch.

[0118] The dataset is: self-built courier human-object interaction behavior dataset, such as Figure 9 As shown in the figure, an Intel RealSense D435i RGBD camera was used to simulate the actual environment of a courier station for video data acquisition. An RGBD multi-source information unpacking action dataset was constructed. The images in this dataset are divided into two action categories: unpacking and unpacking. The images in each category contain elements such as different scenes, different unpacking personnel, different types of boxes, and varying lighting conditions. A total of 400 sets of experimental samples were collected, and each set of experimental samples contained approximately 50-100 frames of color and depth image sequences, which served as the training set for the experiment. At the same time, this example collected 40 sets of image sequences of staff unpacking and unpacking actions in a real post office environment to verify the effectiveness and robustness of the proposed method.

[0119] The experimental steps designed in this embodiment are as follows:

[0120] Step 1: Dataset Partitioning. The 400 sets of simulated scenario experimental samples were divided into UNet model training data and SVM classifier training data in a 1:1 ratio. The 40 sets of experimental samples collected from real scenarios were used as the test sets for UNet and SVM.

[0121] Step 2: UNet model training. Label 200 sets of UNet model training image sequences and train the network model.

[0122] Step 3: SVM data preprocessing. Downsample the SVM training and test image sequences to obtain image subsequences of uniform length of 40 frames. Input the image subsequences into the trained UNet segmentation model to obtain semantically segmented image sequences.

[0123] Step 4: Feature descriptor construction. Extract features from subsequences based on depth and spatiotemporal information to obtain the box depth variance feature vector λ for each subsequence. bn and the hand motion feature vector λ hn , after concatenating the two vectors, we get the eigenvector λ of length 80×1 n ;

[0124] Step 5: SVM data augmentation. The training set data is augmented by adding random Gaussian white noise, resulting in a total of 1000 training set data (80 × 1 × 1000). The test set data is not augmented, resulting in a total of 40 sets (80 × 1 × 40).

[0125] Step 6: SVM classifier training. Input the training set data into the SVM classifier, train the SVM classifier, and recognize human interaction behaviors in a real post office scene.

[0126] Finally, the results of using the SVM classifier to identify the interactive behaviors of people who opened and did not open the boxes at the post office are shown in Table 1:

[0127] Table 1 Recognition accuracy of SVM classifier

[0128]

[0129] Figure 10 It shows some of the opened and unopened image sequences in the post office real scene test set and the SVM recognition results. Figure 10 (a) The sequence is a standard unpacking behavior, and the method of this embodiment determines it as unpacking; Figure 10 In (b), the staff did not open the box for inspection, but only taped it. Since the depth of the box did not change significantly during the process, the method of this embodiment identifies it as unopened. Figure 10 (c) The sequence is interfered by other behaviors, such as the person on the screen is using a mobile phone and the box cannot be detected in the field of view. The method of this embodiment recognizes it as unopened.

[0130] To verify the effectiveness and necessity of each step in the proposed method, this example conducted the ablation experiments shown in Table 2. First, by comparing it with a traditional HSV threshold-based foreground segmentation method (rows 2-3 of Table 2), the UNet-based foreground segmentation method used in this example maintains high segmentation accuracy even when the box and the hand have very similar colors and there is a lot of background noise.

[0131] Figure 11 Describe the box using depth information ( Figure 11 (a)), using RGB information to describe the box ( Figure 11 (b)) Using RGB information to describe human hands ( Figure 11 (c)) and using depth information to describe human hands ( Figure 11 As shown in the feature curve diagram (d), only by using the depth variance (depth information) to describe the box and the motion trajectory (RGB information) to describe the human hand can the unpacking behavior (dashed line) and the non-unpacking behavior (solid line) be well distinguished, thus proving that the feature descriptor based on depth and spatiotemporal information proposed in this embodiment is reasonable and effective.

[0132] In rows 4-6 of Table 2, by comparing with the feature extraction methods using only visible light information and only depth information, it is shown that the feature extraction method based on depth and spatiotemporal information proposed in this embodiment has high accuracy in describing the unpacking and unpacking behaviors in the interactive behaviors of people at the express station. The complementary fusion of RGBD information can significantly improve the recognition accuracy of the SVM classifier.

[0133] Table 2. Comparison results of ablation experiments

[0134]

[0135] In order to verify the robustness of the algorithm proposed in this invention under different interference conditions, this embodiment also conducts interference suppression verification experiments based on different box types, background debris, light changes, and other phenomena in the real unpacking and inspection scene of the post office. The image segmentation results of the UNet model trained in this embodiment are as follows: Figure 12-14 As shown, the SVM classifier character interaction behavior recognition results are calculated for different interference conditions, and the confusion matrix is ​​as follows Figure 15 shown.

[0136] Figure 12 The results of the box type interference experiment are shown, where (a) is the original image, (b) is the segmentation result, (c) is the box color map, and (d) is the human hand color map. Figure 12 It shows that the method proposed in this embodiment can achieve good segmentation of objects and boxes when facing boxes of different types and sizes in real post office scenes, ensuring the accuracy of the final detection results (such as Figure 15 (as shown in (a)).

[0137] like Figure 13 The results of the background clutter interference experiment are shown, where (a) is the original image, (b) is the segmentation result, (c) is the box color image, and (d) is the human hand color image. Figure 13 It shows that when there are common background objects such as mobile phones and tapes that are not conducive to foreground extraction in the field of view, the UNet model trained in this embodiment can still accurately extract the box and the human hand. The confusion matrix of the recognition result is as follows: Figure 15As shown in (b), a relatively high accuracy can still be achieved.

[0138] For 24-hour all-weather monitoring scenarios, changes in light are also one of the interference factors that need to be considered. Figure 14 The results of the light change interference experiment are shown in Figure 1, where (a) is the original image, (b) is the segmentation result, (c) is the box color image, and (d) is the human hand color image. Figure 14 As shown in FIG, the UNet-based semantic segmentation model proposed in this embodiment can achieve accurate segmentation of the box and the human hand under different lighting conditions, and can ensure the accuracy of the final detection result (such as Figure 15 (c)).

[0139] Experimental results demonstrate that the feature descriptors extracted using depth and spatiotemporal information in this embodiment effectively address the diverse interference conditions found in real-world post office inspection scenarios, effectively suppressing interference from various box types, background debris, and lighting variations. The system demonstrates high robustness and accuracy, with both detection precision and recall rates meeting the requirements for identifying interactive human behavior, enabling reliable monitoring of courier workers' unpacking and inspection activities.

[0140] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying interactive behaviors of people in a post office scene, characterized in that: The method comprises: Step 1: Use background subtraction to determine whether there is any behavior in the detection area. If there is behavior in the detection area, the camera collects the behavior video stream, which includes: color image sequence and depth image sequence; Step 2: Perform foreground segmentation based on the UNet network, input the color image sequence into the pre-trained UNet network model, and output the segmentation results with labels to obtain binary images of the box and the human hand; Step 3: Register the depth image and color image; Step 4: Using the box binary image obtained in step 2, extract features of the foreground box in the registered depth image to obtain box depth features; Step 5: Extracting the hand motion trajectory based on the binary hand image obtained in step 2 to obtain the hand motion features; Step 6: Based on the box depth features and hand motion features, construct a feature descriptor containing depth and spatiotemporal information; Step 7: Use the pre-trained SVM classifier to classify the feature descriptors to achieve recognition and detection of human interaction behaviors.

2. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: In step 4, the depth value variance of the box foreground of a single frame image during the unpacking process is used as the feature value representing the image. The calculation formula of the depth variance is: Among them, σ 2 represents the variance of the depth value of the foreground image, n represents the total number of pixels in the foreground image, d k Indicates the depth value of the k-th pixel, d m Represents the average depth value of all pixels in the foreground image; By calculating the variance σ of the depth foreground map of the single frame image box 2 , and then multi-frame image variance σ is converted into 2 Forming the eigenvector λ bn .

3. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: The process of extracting hand motion features in step 5 includes: Extract feature points from the binary image of the hand. Select the hand farthest from the camera and perform pixel traversal on its binary image. Use the row number of the top white pixel in the image as the ordinate and the column number of the leftmost white pixel in the image as the abscissa to obtain the coordinates of the feature point F(x,y). Calculate the pixel distance D from the feature point F(x,y) to the image coordinate origin t , the distance D between the feature points of the continuous frame images and the origin is calculated according to the time sequence t Forming the eigenvector λ hn .

4. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: The process of using the background subtraction method in step 1 to determine whether there is any behavior occurring in the detection area includes: Compare the input image with the background model, and extract and segment the moving target in the image through changes in grayscale, chroma and other features. The calculation formula of the background subtraction method is: d (k) (x,y)=|f (k+1) (x,y)-B (k) (x,y)| Among them, d (k) (x,y) represents the current frame image f( k+1) (x,y) and background image B (k) The difference between (x,y), m (k) (x, y) represents the binary image of the difference image, T represents the threshold of the pixel difference between the current frame image and the background image, and when it is greater than or equal to T, it is judged that there is a foreground target; When there is no moving target entering the camera's field of view, the detection system is in standby mode and only acquires video streams and image frames. The first frame image f0(x, y) is selected as the initial background image B0(x, y). If the scene changes, the background image is updated accordingly. When a moving target appears in the detection area, the binary image m (k) There are a lot of white pixels in (x, y). We calculate the ratio s of white pixels to the total pixels of the image and set the upper limit V of the evaluation standard based on experience. max and lower limit V min ; When s>V min When , it is determined that there is behavior in the unpacking area, and the detection system enters the unpacking detection mode; the previous frame of the current frame is recorded as the starting point Sτ, and Sτ is used as the new background image to replace the initial background image to continue background subtraction until s <V min The behavior is determined to be finished and the current frame is recorded as the end point E τ , S τ to E τ This video frame is treated as a suspected unboxing behavior and subsequently processed; If s is always less than V min or greater than V max , it is judged that there is a sudden change in illumination in the current frame or no behavior occurs in the detection area, and the judgment of the next frame is continued.

5. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: The process of registering the depth image and the color image in step 3 includes: First, restore the pixel points of the depth map to the depth coordinate system, and directly obtain the Euclidean transformation matrix T from the depth coordinate system to the color coordinate system d2c , and then map the depth point in the color coordinate system to the color plane of Z = 1. The conversion formula is: Among them, T d2c Represents the transformation matrix from the depth coordinate system to the color coordinate system, T w2c Represents the transformation matrix from the world coordinate system to the color coordinate system, T w2d Represents the transformation matrix from the world coordinate system to the depth coordinate system.

6. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: During the feature extraction process, the UNet network gradually downsamples the feature map through 3×3 convolution, ReLU activation and 2×2 maximum pooling, gradually reducing the input samples while increasing the number of channels to compensate for the loss of resolution.

7. The method for identifying interactive behaviors of people in a post office scene according to claim 1, characterized in that: During the upsampling process, the UNet network performs 2×2 transposed convolution and 1×1 convolution on the feature map. After each upsampling, it is fused at the same scale as the number of channels corresponding to the feature extraction part, gradually restoring the image details, so that the size of the feature map gradually increases and the number of channels gradually decreases, and finally outputs a semantic segmentation image.

8. A post office scene character interaction behavior detection system, characterized by: The detection system is used to implement the method for identifying interactive behaviors of people in a post office scene according to any one of claims 1 to 7, comprising: Video stream acquisition equipment, used to collect behavior video streams within the detection area; A behavior trigger module, which uses background subtraction to determine whether there is any behavior occurring in the detection area, and if so, triggers the video stream acquisition device to acquire the video stream; The UNet foreground segmentation module is used to perform semantic segmentation on the images in the captured video stream and extract the box and human hand images in the foreground moving objects; The feature extraction module extracts features from the box and hand images after foreground segmentation to obtain box depth features and hand motion features; The classification output module uses the SVM classifier to identify and classify the extracted box depth features and hand motion features, and outputs the detection results.

9. The post office scene character interaction behavior detection system according to claim 8, characterized in that: The video stream acquisition device includes: an RGBD camera.

Images