A method for recognizing human interaction behavior in a restaurant scene and a security monitoring system

Abstract

The application discloses a kind of restaurant scene under the identification method and safety monitoring system of character interactive behavior, belong to video analysis technical field, the method includes: determine each video monitoring collection area picture, data pre-processing etc.;Establish restaurant staff face and human target, customer human body, restaurant common article label library;Establish the label library of the body related action behavior of interaction with restaurant common article and the interaction between people, judge the safety / unsafe of its behavior and establish label;Build double-task double-flow parallel learning network, extract the feature information of target identification and the feature information of character interactive action behavior;Process the feature information extracted by fusion model network, build a network model for restaurant customer safety monitoring and employee service work content understanding, and judge whether there is unsafe employee and customer behavior, establish a kind of intelligent monitoring system, and have certain help effect to the automation level of video intelligent monitoring system.

Description

TECHNICAL FIELD

[0001] The application discloses a person interactive behavior recognition method and a safety monitoring system in a restaurant scene, and belongs to the technical field of video analysis. BACKGROUND

[0002] At present, some bad picketing, violence and beating events occur in public places, especially in some restaurants, which leads to the fact that dining safety cannot be guaranteed, and a safety warning is sent to people in public places. Therefore, it is very important and necessary to establish a person interactive behavior recognition method and a safety monitoring system in a restaurant scene, create a safe and good restaurant dining environment, and thus supervise and alarm the behaviors of customers and staff in the restaurant in real time.

[0003] At present, many behavior recognition methods at home and abroad focus more on single and multi-person behaviors, single and multi-frame image behaviors. According to different data sets used and different input data types, the current mainstream interactive behavior recognition methods can be divided into two categories: visible light (RGB) information-based and depth (RGB-D) information-based. According to different data types of input images, the current mainstream interactive behavior recognition methods can be divided into two categories: single-frame image (RGB) interactive behavior action recognition and multi-frame (RGB-T) continuous interactive behavior action recognition. According to different recognition human body parts, the recognition methods can also be divided into two categories: interactive action recognition based on human torso state and interactive behavior recognition based on each limb behavior semantics. These methods all need a large amount of pre-training processing and expensive GPU consumption, and the obtained model can only achieve a certain accuracy in the recognition of general interactive behaviors, and the recognition robustness in the actual scene is very insufficient.

[0004] However, in actual scene use, it is required to recognize the effect to meet certain accuracy and robustness requirements, avoid high training cost, and meet the requirements of small delay and real-time speed. No matter directly using any model and any data set for landing application, all the above methods cannot meet all the requirements, and the above methods cannot give a good reference and solution. SUMMARY

[0005] In order to meet the requirements of the current person interactive behavior recognition scheme in the restaurant safety monitoring scene, the person interactive behavior recognition method and the safety monitoring system in the restaurant scene can actually, accurately and quickly detect the person interactive behavior, and classify and distinguish the behavior safety rationality. The application provides a person interactive behavior recognition method and a safety monitoring system in a restaurant scene, and the technical scheme is as follows.

[0006] The application aims to provide a person interactive behavior recognition method and a safety monitoring system in a restaurant scene, and the method comprises the following steps.

[0007] Step S1: First, determine the video surveillance acquisition area, the installation location of the shooting equipment, and confirm whether the installation conditions are met. Then, extract frames from the acquired video footage and obtain single-frame images with equal time step intervals.

[0008] Step S2: Establish a label library for restaurant employee faces and human targets, customer human bodies, and common restaurant items and tools, and build a model that can identify human instances and object targets, thereby enabling the extraction of target detection-related features.

[0009] Step S3: Establish a tag library for human body movements and actions related to interactions with common restaurant items and tools, as well as for human-to-human interactions. This includes tags for human body movements and tags for human / object / human interactions. Simultaneously, evaluate these actions and behaviors to determine their safety / unsafety and rationality / irrationality. Establish rationality and safety tags and construct a model that can identify human interactions within the restaurant, thereby enabling the extraction of features related to human interactions.

[0010] Step S4: Then, the acquired video footage is processed by region, different regions and security levels are set on the image, and a multi-task dual-stream parallel learning network is constructed. The features of different (two) task targets are regarded as features of different (two) modalities to create a dual-stream backbone. For two types of human targets (dining customers, service personnel) and multiple types of object targets in the scene that may interact with people, the feature information of target recognition and the feature information of human interaction behavior are extracted.

[0011] Step S5: Process the human target object features and human interaction behavior features extracted from the target area by the fusion model network, and construct a deep neural network model for monitoring restaurant customer safety and understanding and reasoning about employee service work content. Use identification labels based on customer safe / unsafe behavior and identification labels based on employee work content to classify and identify whether there is unsafe or unreasonable behavior in the current scene, so as to ensure customer dining safety and confirm employee service content.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] S11: First, determine the video monitoring collection area, installation position of the shooting device and confirm whether the installation conditions are met: that is, to determine the restaurant interior monitoring device needs to be aligned to capture the area, in general, the restaurant customer dining area, public service area, health responsibility area in front of the door, etc. are responsible for and need to be monitored for security; and in order to improve the video collection range of a single camera, each camera should be installed as high as possible and monitored from a bird's eye view; at the same time, no matter which position the camera is shooting, the quality of the shooting picture should be met, that is, the picture is clear and visible under any lighting at any business time and the size of the figure is appropriate.

[0014] S12: Secondly, frame the collected video pictures and obtain single-frame images with equal time steps: analyze the behavior and action video collected by each camera for 10 minutes, and uniformly divide the 10-minute video into 200 non-overlapping 3-second segments, and extract 9 clear segment frame images at the beginning, middle and end of the 3-second action segment. Preprocessing, without losing image information, uniformize the size of the above picture data set, using the data calculated by ImageNet training set sampling: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] where mean represents the mean, std represents the variance; normalize; since the unsafe and unreasonable human interaction behavior in the daily collected video is only a small number of cases, which may cause a very large number of differences between the positive and negative samples of the collected video data, resulting in a serious imbalance problem of positive and negative samples. To solve this problem, when collecting video pictures, some video picture data of unsafe and unreasonable human interaction behavior that is not common in daily life needs to be artificially intervened, that is, the corresponding behavior in the monitoring video picture is artificially performed: such as harassment and infringement, physical assault, drunkenness and fall, unstable walking and other customer abnormal behaviors; similarly, the incorrect and bad work behavior of service staff is also collected and processed into the video picture data set that needs to be trained and tested.

[0015] Further, the step S2 specifically includes the following steps:

[0016] S21: Establish a restaurant staff face target label library, and construct a staff face target label according to the Celeba face data set labeling rule; the specific steps include: using the OpenCV self-made program to batch label the non-overlapping images collected in step S12, that is, using a mouse and a keyboard to sequentially frame, affine transform and text label on the collected images, and finally obtaining the jpg format images with the face centered, cropped and unified size of 178x178; the annotation file is 5 txt text files, and each object is labeled with one line of text: 1, identity specifies the face label corresponding to each picture, and the format is picture name, face ID; 2, list_attr specifies the face attribute, and the original format is 40 categories, and we select face attributes such as whether the face has black hair, whether it wears glasses, male and female according to the actual situation; 3, list_bbox labels the position of the face in the picture, and the format is picture name bounding box four element information, that is, left upper corner x axis coordinate, y axis coordinate, width, height; 4, list_landmarks_align labels the position of the face key point in the centered picture, and there are 5 key points, that is, eyes, nose and corners of the mouth, and the format is picture name, left eye x axis coordinate, y axis coordinate, right eye x axis coordinate, y axis coordinate, nose x axis coordinate, y axis coordinate, left corner of the mouth x axis coordinate, y axis coordinate, right corner of the mouth x axis coordinate, y axis coordinate; 5, list_landmarks labels the key point position of the face in the original picture, and the format is the same as above.

[0017] S22: Establish an employee body target, customer body target and restaurant common tool label library, and construct an employee body target, customer body target and restaurant common tool label according to the YOLO target detection data set labeling rule; the specific steps include: using the OpenCV self-made program or the existing Labelimg software to frame, affine transform and text label on the collected images one by one, and finally obtaining two folders: one is a folder storing original pictures, and the other is a label text folder; wherein, the label text folder stores a txt text file named with the picture name, and each object is labeled with one line of information, and the labeling format is class ID, center point x axis coordinate, center point y axis coordinate, target width, target height, but it needs to be noted that whether the coordinate or the width and height needs to be normalized, that is, the target center coordinate in the picture and the target height and width percentage of the picture need to be defined in percentage form. And a class classes label text file is also needed, and the class name is labeled in each line according to the class ID sequence. The classes here include customer body, working service personnel body, common tools in restaurant and general tools that may appear.

[0018] Further, the step S3 specifically comprises the following steps:

[0019] S31: Establish the action behavior of the character interacting with the common objects and tools in the restaurant, and label the data according to the primitive feature form of the target limb state of the character in HAKE (Human Activity Knowledge Engine), that is, use 93 different limb primitive features of 6 limb parts in part_state_93_new to label the customers and staff; and at the same time, use 157 different current behavior actions of the character to label the character, save as a single mdb file, and the labeling format is picture path, action sequence number, character bbox position, primitive sequence number of each limb of the character, flag of groundtruth, and at the same time, construct positive and negative samples, and remark with gt_flag.

[0020] S32: Establish the label library of the action behavior of the character interacting with the common objects and tools in the restaurant, and the interaction action between people, that is, the label of the action of the character and the label of the interaction behavior of people / objects / people, and label according to the label form of HOI Classification and HOI Detection in HICO (Humans Interacting with Common Objects). First, HOI Classification: taking the objects created in S22 and the action of the character created in S31 as targets, a set of binary labels is established, each label represents the existence or nonexistence of an HOI class, and "ambiguous" samples are created at the same time as the positive and negative samples; secondly, integrate the detection object edge created in S22 and the target bounding box of the character created in S31, each bounding box locates a person and an object, or a person and another person, and finally creates an HOI class label. Both types of annotation files are mat files, the HOI Classification annotation file includes: a one-dimensional ndarray with a length of the number of interaction action categories, for each category, there are picture file path, interaction action name, synonym, etc. attributes; training and testing picture name list; training and testing set labels and their flag categories; and the HOI Detection annotation file includes: the path of each picture, the image size, the character interaction information; wherein the character interaction information includes action sequence number, character bbox (array, multiple), object bbox (array, multiple), interaction task number pair, etc.; finally, create the annotation information list_actions of all the objects and their gerundive combination categories under the restaurant scene.

[0021] S33: Evaluate the task interaction action behavior in S32, judge the safety / unsafe and reasonable / irrational of its behavior, and establish the reasonable safety label; as described in S12, the negative sample video segment pictures added by human and existing in the collected video are incorrect, bad working behaviors of the staff and unsafe, unreasonable behaviors of the customers, which need to add this safety rationality label in S32 among all the existing interaction action attributes, that is, a two-level data flag to represent.

[0022] Further, the step S4 specifically includes the following steps:

[0023] S41: Perform regional processing on the collected video pictures, and set different regions and safety levels on the image: according to the size of the restaurant dining area and the overall scale of the restaurant, there can be two cases. One case is a small restaurant with a small scale: a single video monitoring camera can be used to monitor the overall restaurant from a high position to a certain extent without dead angles. The other case is a large restaurant with a very large scale: the range of the same dining area is too large, and a single monitoring camera cannot completely monitor the overall area, so multiple cameras are used for auxiliary shooting to complete the monitoring. Or there are more cameras for monitoring multiple floor areas, etc. No matter which case, there is a division of the dining area and a division of the safety level of each area according to the specific situation of the restaurant: a single camera divides the dining area, adds a region mask, sets the customer ROI area and the staff ROI area, and performs targeted recognition, reasoning and prediction; while multiple cameras assist each other, there is a problem of region overlap, and the identification, reasoning and prediction of the same area need to divide the region names of each camera image, and according to the situation, whether the extracted information needs to be fused needs to be selected; in addition, multi-layer and multi-region also need to consider whether to integrate cross-image information, perform more complex subsequent operations such as person re-identification and person interaction behavior re-identification, so as to establish a more large and complex, overall large-scale restaurant personnel and behavior safety system.

[0024] S42: Construct a multi-task (double-task) double-flow parallel learning network, treat the features of different (two) task targets as features of different (two) modalities to create a double-flow backbone, and extract the feature information of target recognition and the feature information of person interaction action behavior for the two types of human targets (dining customers, working service personnel) in each image, multiple types of object targets that can interact with people in the scene, etc. However, the same type of encoder is used for extraction: one branch encodes to extract object and human target detection features, and the other branch encodes to extract limb pasta features and person interaction action features.

[0025] Further, step S42 specifically includes the following steps:

[0026] S421: First, select the intermediate frame picture of each type of character interaction action collected and processed in S12, and take two video image pictures at the same time frame of the same interaction behavior as the input of the double-flow network.

[0027] S422: The first branch of the double-flow network uses the Swifted Window Transformer encoder structure, and the window size is 7x7. The parameters used are part of the Swin-B structure parameters, that is, the encoder structure is divided into 4 layers stage, the hidden layer feature dimension of the first stage is 128, the number of continuous Swin-Tranformer blocks of each stage is 2, 2, 18, and 2, respectively, and the number of heads when doing window attention mechanism operation is 4, 8, 16, and 32, respectively. This part is used to extract object and human target detection features and is used as a backbone.

[0028] S423: Similarly, the second branch of the double-flow network also uses the Swifted Window Transformer encoder structure, but the parameters used are Swin-B structure parameters. In addition, the number of heads when doing window attention mechanism operation is changed to 3, 6, 12, and 24, respectively, and the rest of the parameters remain unchanged. This part is used to extract limb pasta features and character interaction action features, and is also used as a backbone.

[0029] S424: No matter which branch encoder, it is known that the input of a batch (batch number B) of pictures img has a dimension of (B, H, W, 3): first, convert it from two-dimensional image data to patch encoding form, that is, use stride=4, 4x4 convolution kernel to downsample patch size (uniformly set to 4) multiple, and the input channel number is converted from 3 to the encoding feature dimension (all set to 128). Then, flatten the feature map of each patch after downsampling to one dimension and adjust the tensor size, and then perform layer normalization operation. The img data size changes to (B, tensor data of (B, 128); next, create the BasicLayer of stage1, stage2, stage3, stage4 respectively according to the depth of 2, 2, 18, 2 and the number of heads of 4, 8, 16, 32 (3, 6, 12, 24): wherein the BasicLayer includes the Swin-Transformer block which specifies the depth (number) and the number of heads in the window-based multi-head attention calculation, and a Patch Merging module which halves the feature map size each time and doubles the number of channel dimensions, but does not contain the Patch Merging module in the last stage because the 4 times of down-sampling calculation have been completed; therefore, the size dimensions of the previous tensor data change to (B, , 256), (B, , 512), (B, , 1024) and the like in turn after passing through each BasicLayer; further, the calculation process of the Swin-Transformer block is shown in the following formulas (1)-(4):

[0030] (1)

[0031] (2)

[0032] (3)

[0033] (4)

[0034] Further, the core steps of the calculation process are W-MSA and SW-MSA, the former is the calculation of multi-head self-attention mechanism under the whole feature map of MSA, that is, shown in formula (5):

[0035] (5)

[0036] The former takes the whole feature map as the receptive field; while in the latter W-MSA, only the multi-head self-attention mechanism in a single window size is calculated, the receptive field is greatly reduced, and the information interaction between the windows is also lacking; therefore, SW-MSA is designed, which divides the whole feature map into 9 blocks of 4 different sizes, and defines 4 blocks of regions, changes the region position through translation transformation, adds Mask mask to calculate the attention results of the originally unrelated regions, and changes the result -100 to negative, to 0 after SoftMax operation, waits for the end of the calculation to move the region position back to the previous position, and the bias in the attention mechanism introduces the relative position bias, here a new relative position bias table is introduced, which is learned during training, and the bias of any patch to other patches is directly taken from the information learned from the network.

[0037] S425: After the construction of the dual-flow branch backbone, the features output by the dual-flow branch backbone at the same layer need to be operated subsequently. After the continuous Swin-Transformer block modules in each stage are operated, the product operation is performed on the 4-scale features obtained, that is, after each down-sampling calculation and window multi-head self-attention calculation, the features from the same picture but different identification tasks are aligned to show the common significant position. The features of the two branches are aligned in the control part, and then the respective channel parts are recalibrated. The common spatial attention calculation is shown in equation (6):

[0038] (6)

[0039] wherein, and are the two branch features that need to be aligned; then the common spatial attention map is used as the weight of the two task features, and the spatial alignment of the two modes is realized in the following way, as shown in equations (7) and (8):

[0040] (7)

[0041] (8)

[0042] Then, the aligned features in the spatial part , are respectively executed spatial attention to generate a spatial attention map, and the spatial attention calculation is shown in equation (9):

[0043] (9)

[0044] The figure shows more weight on more significant content of each task feature, and then multiplies each channel attention map with the original feature to achieve channel recalibration, as shown in equations (10), (11):

[0045] (10)

[0046] (11)

[0047] Finally, each layer of stage integrates the two task features into the calibrated fusion features as shown in equation (12):

[0048] (12)

[0049] S426: Then the task 1 in the dual-stream network needs to be constructed, that is, the subsequent neck network of the object and human target detection feature extraction task, which uses Path Aggergation Network (PANet) and Feature Pyramid Network (FPN). FPN conveys semantic feature information from top to bottom, and PAN conveys positioning information from bottom to top; from different backbone layers, position information and semantic feature information are extracted from different feature layers using multiple scales: pictures 、 、 Under three different scale conditions, the identification and position prediction of large, medium and small human targets and object targets are realized respectively.

[0050] S427: Then construct task 2 in the dual-stream network, that is, the heads part of extracting limb pasta features and human interaction action features, which uses MLP with residual connection to predict human limb state and interactive action for the output of the Swin-transformer encoder of the entire task 2, and fuse and classify it with the target recognition features in S426, and finally obtain the human interaction behavior between all detected targets, that is, two types of interactive information output of human target and object target: human-action-object and human-action-human. Among them, the pasta output uses the prior wordmap knowledge base pasta_language_matrix.npy of Resnet50 layer4 and HAKE-meta primitive reasoning to increase the recommendation accuracy and speed, and finally outputs the interactive action information between each human.

[0051] S43: Construction of loss function and implementation of training process:

[0052] S431: First, the detection of the target of the character and the target of the object in the dual task, the loss function is generally composed of two parts of classification loss and regression loss, that is, to predict the bbox frame and the target category. The classification loss function uses Focal loss, which is a loss function for handling class imbalance and difficult sample mining. The calculation formula is shown in equation (13):

[0053] (13)

[0054] wherein, is the naked result predicted by the model and the probability value after softmax, is the calculation result of the cross-entropy loss, so the is smaller, the prediction is less accurate, and this item will increase, so this item is also used as the coefficient of the difficult sample. The more inaccurate the prediction, the more the Focal loss tends to regard the sample as a difficult sample, and the larger the coefficient. The purpose is to make the difficult sample contribute more to the loss and gradient. The is the category weight coefficient. If you have a class- imbalanced dataset, you certainly want to assign a high weight to the loss contribution of the class with fewer quantities. The plays such a role. Therefore, should be a vector, and the length of the vector is equal to the number of categories, which is used to store the weight of each category.

[0055] The regression loss function uses CIoU Loss to measure the loss of the bbox, which considers the loss of the boundary box length and width, increases the similarity between the predicted frame and the real frame, and can converge faster. The calculation formula is shown in equation (14):

[0056] (14)

[0057] wherein, is a regularization term introduced from DIoU, which also has the effect of preventing overfitting; α is a positive trade-off parameter, and ν is used to measure the consistency of the length-width ratio.

[0058] S432: Secondly, the character interaction behavior recognition task in the dual task, which recognizes three parts of the target recognition content and the interactive action recognition content, and then merges to recognize a complete character-verb-object or character-verb-character two types of interactive information. Each target here uses the Focal loss function in S431. The specific calculation process is the same as above.

[0059] S433: The specific training process adopts a distributed training method, stores data on two or more GPUs for training, builds an optimizer, iteratively trains, adjusts network parameters until the model converges, and uses pre-trained two-branch backbone network weights to speed up the training process.

[0060] Further, the step S5 specifically comprises the following steps:

[0061] S51: Process the human target object features and human interaction behavior features extracted by the fusion model network to construct a deep neural network model for understanding and reasoning the safety monitoring of restaurant customers and the work content of employees: After the operation and processing of S4, the target detection information and human interaction behavior information are extracted from the picture. A deeper classifier is constructed to judge the target behavior: customer action safety / non-safety behavior judgment and employee work content correctness judgment; here, the labels made by S33 are used, and the AdaBoost weak learner combination reasoning is used to judge the customer behavior and employee work behavior.

[0062] S52: Finally, a complete safety monitoring system is constructed, i.e. in addition to the process of equipment installation, algorithm calculation, model training, etc., a complete monitoring system is also needed: i.e. processing the pictures taken by the real-time monitor or the pictures taken by the monitoring camera previously, after the algorithm calculation of the model gets the recognition result, the system will classify and judge according to the result information, and then selectively verify according to the judgment result and process the current or previous behavior, see the specific process in the attached Figure 2 . BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0064] Figure 1 The framework flowchart of the restaurant scene human interaction behavior recognition method embodiment of the present application.

[0065] Figure 2 The safety monitoring system flowchart of the restaurant scene human interaction behavior recognition embodiment of the present application.

[0066] Figure 3 The model network structure diagram of the restaurant scene human interaction behavior recognition method of the present application. DETAILED DESCRIPTION

[0067] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0068] In order to make the purpose, technical scheme and advantages of the present application more clear, the following will combine the drawings of the present application with the specific embodiments of the present application. Figure 1 、 2 The person interactive behavior recognition method, device and security monitoring system provided by the present application will be further described in detail.

[0069] A person interactive behavior recognition method in a restaurant scene, comprising the following steps:

[0070] Specifically, in the present embodiment, the interactive behavior in the applied scene specifically refers to the behavior of customers and restaurant staff in a restaurant scene.

[0071] S1: First, determine the video monitoring collection area, the installation position of the shooting device, and confirm whether the installation conditions are met, then frame the collected video pictures and obtain single-frame images at equal time step intervals; in the present embodiment, the monitoring collection site is set to the first floor of a restaurant, a Canon EOS M6 Mark2 camera is used, and a support is used to elevate the camera to a height higher than that at 11:30 to 13:30 and 17:00 to 19:00, two time periods, to shoot the dining and shopping scenes, and the shooting video follows three principles of overlooking shooting without blind area, avoiding shaking of the shooting device, and ensuring that the size of the characters in the picture is appropriate. In order to solve the problem of uneven positive and negative samples, when collecting video pictures, some unsafe and unreasonable person interactive behaviors that are not common in daily life, i.e. artificially performed behaviors in the monitoring video pictures, need to be artificially intervened and added: such as abnormal behaviors of customers harassing and infringing, physically attacking, etc.; similarly, incorrect and undesirable working behaviors of service personnel are also collected and processed into the video picture data set for training and testing. Finally, 50 10-minute videos, 10,000 non-overlapping 3-second video segments, and a total of 30,000 picture data are shot, preprocessed, and the picture data set size is unified under the premise of not losing image information, and the data calculated using the ImageNet training set sampling: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] are normalized, wherein mean represents the mean, and std represents the variance.

[0072] S2: Establish a part of employee face and human body target, customer human body, and canteen common article tool label library, and construct a model capable of recognizing person instances and object targets, so as to extract target detection related features. The specific operation is as above.

[0073] S21: Establish a restaurant staff face target label library, and construct a staff face target label according to the Celeba face data set labeling rule; the specific steps include: using the OpenCV self-made program to batch label the non-overlapping images collected in step S12, that is, using a mouse and a keyboard to sequentially frame, affine transform and text label on the collected images, and finally obtaining the jpg format images with the face centered, cropped and unified size of 178x178; the annotation file is 5 txt text files, and each object is labeled with one line of text: 1, identity specifies the face label corresponding to each picture, and the format is picture name, face ID; 2, list_attr specifies the face attribute, and the original format is 40 categories, and we select face attributes such as whether the face has black hair, whether it wears glasses, male and female according to the actual situation; 3, list_bbox labels the position of the face in the picture, and the format is picture name bounding box four element information, that is, left upper corner x axis coordinate, y axis coordinate, width, height; 4, list_landmarks_align labels the position of the face key points in the centered picture, a total of 5 key points, that is, eyes, nose and corners of the mouth, and the format is picture name, left eye x axis coordinate, y axis coordinate, right eye x axis coordinate, y axis coordinate, nose x axis coordinate, y axis coordinate, left corner of the mouth x axis coordinate, y axis coordinate, right corner of the mouth x axis coordinate, y axis coordinate; 5, list_landmarks labels the key point position of the face in the original picture, and the format is the same as above.

[0074] S22: Establish an employee body target, a customer body target and a restaurant common article tool label library, and construct an employee body target, a customer body target and a restaurant common article tool label according to the YOLO target detection data set labeling rule; the specific steps include: using the OpenCV self-made program or the existing Labelimg software to frame, affine transform and text label on the collected images one by one, and finally obtaining two folders: one is a folder storing original pictures, and the other is a label text folder; wherein, the label text folder stores a txt text file named with a picture name, each object is labeled with one line of information, and the labeling format is class ID, center point x axis coordinate, center point y axis coordinate, target width, target height, but it needs to be noted that whether the coordinate or the width and height needs to be normalized, that is, the target center coordinate in the picture and the target height and width percentage of the picture need to be defined in the form of percentage. And a class classes label text file is also needed, and the class name is labeled in each line according to the class ID sequence. The classes here include customer body, working service staff body, common articles in restaurant and general articles that may appear.

[0075] S3: Establish the action behavior of the character's limbs related to the interaction with the common food tool in the canteen, the label library of the interaction action between people, that is, the action label of the character's limbs and the action label of the interaction between people, and evaluate these action behaviors to judge the safety / unsafety and rationality / irrationality of the behaviors, establish the reasonable safety label, and construct a model capable of identifying the interaction action behavior of the characters in the canteen, so as to extract the features related to the interaction behavior of the characters.

[0076] S31: Establish the action behavior of the character's limbs related to the interaction with the common food tool in the restaurant, and perform data labeling according to the primitive feature form of the character target limb state in HAKE, that is, use 93 different limb primitive features of 6 limb parts in part_state_93_new to label the customers and staff; and at the same time, use 157 different current behavior actions of the characters to label the characters, save as a single mdb file, and the labeling format is picture path, action sequence number, character bbox position, primitive sequence number of each limb of the character, flag mark of groundtruth, and construct positive and negative samples with gt_flag.

[0077] S32: Establish the label library of the human action behavior interacting with the restaurant common object tool and the human-human interaction action, i.e. the human body action label and the human / object / human interaction behavior label, according to the label form of the two aspects of HOI Classification and HOI Detection in HICO (Humans Interacting with Common Objects). First, HOI Classification: taking the object created in S22 and the human action behavior created in S31 as the target, a set of binary labels is established, each label represents the existence or nonexistence of an HOI class, and the positive and negative samples are created at the same time, and "ambiguous" samples are also created, i.e. irrelevant interaction samples; second, the detection object edge created in S22 and the human target bounding box created in S31 are integrated, each bounding box locates a person and an object, or a person and another person, and finally a HOI class label is created. Both types of annotation files are mat files, the HOI Classification annotation file includes: a one-dimensional ndarray with a length of the number of interaction action categories, for each category, there are picture file path, interaction action name, synonym, etc. attributes; training and testing picture name list; training and testing set labels and flag categories; while the HOI Detection annotation file includes: the path of each picture, image size, human interaction information; among which the human interaction information includes action sequence number, human bbox (array, multiple), object bbox (array, multiple), interaction task number pair, etc. Finally, the annotation information list_actions of all the objects and their gerundive combination categories built before under the restaurant scene needs to be created.

[0078] S33: Evaluate the task interaction action behavior in S32, judge its behavior safety / unsafe and rationality / irrationality, and establish the rationality and safety label; as described in S12, the negative sample video segment pictures added by human and existing in the collected video are all incorrect and bad work behaviors of the staff and unsafe and irrational behaviors of the customers, which need to add this safety rationality label to all the existing interaction action attributes in S32, i.e. a binary data flag to represent.

[0079] S4: Then the collected video pictures are processed in different regions, different regions and security levels are set on the image, such as customer dining area, customer dining area, canteen staff dining area, etc. A multi-task double-flow parallel learning network is constructed, different (two) task features are regarded as different (two) modal features to create a double-flow backbone, and the feature information of target recognition and the feature information of human interactive action behavior are extracted for the two types of human targets (dining customers, working service personnel) in each image, objects that may interact with people in the scene, etc. An experimental network model is built on a laboratory server with dual 3090 graphics cards using pytorch.

[0080] S41: The collected video pictures are processed in different regions, different regions and security levels are set on the image: according to the size of the restaurant dining area and the overall scale of the restaurant, there may be two cases. One case is a small restaurant with a small scale: a single video monitoring camera can be used to monitor the overall restaurant from a high position to a certain extent without dead angles; the other case is a large restaurant with a very large scale: the range of the same dining area is too large, and a single monitoring camera cannot completely monitor the overall area, so multiple cameras must be used for complete monitoring. Or there are more cameras in the restaurant to monitor multiple floors, etc. No matter which case, there is a division of dining areas and a division of security levels for each area according to the specific situation of the restaurant: a single camera divides the dining area and adds a region mask to set the customer ROI area and the staff ROI area, and targeted recognition, reasoning and prediction are performed; while multiple cameras assist each other, there is a problem of region overlap, and the identification, reasoning and prediction of the same area need to be divided into area names for each camera image, and whether the extracted information needs to be fused is selected according to the situation; in addition, multi-layer and multi-region also need to consider whether to integrate cross-image information for more complex subsequent operations such as person re-identification and person interactive behavior re-identification, so as to establish a more complex and large-scale restaurant personnel and behavior security system.

[0081] S42: A multi-task double-flow parallel learning network is constructed, different (two) task features are regarded as different (two) modal features to create a double-flow backbone, and the feature information of target recognition and the feature information of human interactive action behavior are extracted for the two types of human targets (dining customers, working service personnel) in each image, objects that may interact with people in the scene, etc. However, the same type of encoder is used for extraction: one branch encodes and extracts object and human target detection features, and the other branch encodes and extracts limb pasta features and human interactive action features.

[0082] S421: First, select the intermediate frame picture of each type of character interaction action collected and processed in S12, and take two video image pictures at the same time frame of the same interaction behavior as the input of the double-flow network.

[0083] S422: The first branch of the double-flow network uses the Swifted Window Transformer encoder structure, and the window size is 7x7. The parameters used are part of the Swin-B structure parameters, that is, the encoder structure is divided into 4 layers stage, the hidden layer feature dimension of the first stage is 128, the number of continuous Swin-Tranformer blocks of each stage is 2, 2, 18, and 2 respectively, and the number of heads when doing window attention mechanism operation is 4, 8, 16, and 32 respectively. This part is used to extract object and human target detection features, and is used as a backbone.

[0084] S423: Similarly, the second branch of the double-flow network also uses the Swifted Window Transformer encoder structure, but the parameters used are Swin-B structure parameters. In addition, the number of heads when doing window attention mechanism operation is changed to 3, 6, 12, and 24 respectively, and the rest of the parameters remain unchanged. This part is used to extract limb pasta features and character interaction action features, and is also used as a backbone.

[0085] S424: No matter which branch encoder, given an input of a batch (batch number B) of pictures img with a dimension of (B, H, W, 3): first, convert it from two-dimensional image data to patch encoding form, that is, use stride=4, 4x4 convolution kernel to downsample patch size (uniformly set to 4) multiple, while the number of input channels is converted from 3 to the encoding feature dimension (all set to 128), then flatten the feature map of each patch after downsampling to one dimension and adjust the tensor size, and then perform layer normalization operation, the img data size changes to (B, tensor data of (B, 128); next, create the BasicLayer of stage1, stage2, stage3, stage4 respectively according to the depth of 2, 2, 18, 2 and the number of heads of 4, 8, 16, 32 (3, 6, 12, 24): wherein the BasicLayer includes the Swin-Transformer block which specifies the depth (number) and the number of heads in the window-based multi-head attention calculation, and a Patch Merging module which halves the feature map size each time and doubles the number of channel dimensions, but does not contain the Patch Merging module in the last stage because the 4 times of down-sampling calculation have been completed; therefore, the size dimensions of the previous tensor data change to (B, , 256), (B, , 512), (B, , 1024) and the like in turn after passing through each BasicLayer; further, the calculation process of the Swin-Transformer block is shown in the following formula:

[0086]

[0087]

[0088]

[0089]

[0090] Further, the core steps of the calculation process are W-MSA and SW-MSA, the former is the calculation of multi-head self-attention mechanism under the whole feature map of MSA, that is, as shown in the following formula:

[0091]

[0092] The former takes the whole feature map as the receptive field; while in the latter W-MSA, the multi-head self-attention mechanism is calculated only in a single window size, the receptive field is greatly reduced, and the information interaction between windows is also lacking; therefore, SW-MSA is designed, which divides the whole feature map into 9 blocks of 4 different sizes, and defines 4 blocks of regions, changes the region position by translation transformation, adds Mask mask to calculate the attention results of the originally unrelated regions, and changes the result -100 to negative, to 0 after SoftMax operation, waits for the end of the calculation to move the region position back to the previous position, and the bias in the attention mechanism introduces the relative position bias in the calculation process, here a new relative position bias table is introduced, which is learned during training, and the bias of any patch to other patches is directly taken from the information learned from the network.

[0093] S425: After the construction of the dual-flow branch backbone, the features output by the dual-flow branch backbone at the same layer need to be operated subsequently. The four scale features obtained after the continuous Swin-Transformer block modules in each stage are multiplied, that is, after each down-sampling calculation and window multi-head self-attention calculation, the features from the same picture but different identification tasks are aligned to show the common significant position. The features of the two branches are aligned and then the respective channel parts are recalibrated in the control part. The common spatial attention calculation is as follows:

[0094]

[0095] wherein, and are the two branch features that need to be aligned; then the common spatial attention map is used as the weight of the two task features, and the spatial alignment of the two modes is realized as follows:

[0096]

[0097]

[0098] Then, the aligned features in the spatial part , are respectively executed spatial attention to generate a spatial attention map, and the spatial attention calculation is as follows:

[0099]

[0100] The figure shows more weight on more significant content of each task feature, and then multiplies each channel attention map with the original feature to achieve channel recalibration, as shown in the following formula:

[0101]

[0102]

[0103] Finally, each layer of stage integrates the two task features into the calibrated fusion features as follows:

[0104]

[0105] S426: Then the task 1 in the dual-stream network needs to be constructed, that is, the subsequent neck network of the object and human target detection feature extraction task, which uses Path Aggergation Network (PANet) and Feature Pyramid Network (FPN). FPN conveys semantic feature information from top to bottom, and PAN conveys positioning information from bottom to top; from different backbone layers, position information and semantic feature information are extracted from different feature layers using multiple scales: pictures 、 、 Under three different scale conditions, large, medium and small, the identification and position prediction of human targets and object targets are realized respectively.

[0106] S427: Then build task 2 in the dual-stream network, that is, the heads part of extracting limb pasta features and human interaction action features, which uses MLP with residual connection to predict human limb state and interactive action for the output of the Swin-transformer encoder of the entire task 2, and fuse and classify it with the target recognition feature in S426, and finally get the human interaction behavior between all detected targets, that is, two types of interactive information output of human target and object target: human-action-object and human-action-human. Among them, the pasta output uses the prior wordmap knowledge base pasta_language_matrix.npy of Resnet50 layer4 and HAKE-meta primitive reasoning to increase the recommendation accuracy and speed, and finally outputs the interactive action information between each human.

[0107] S43: Construction of loss function and implementation of training process:

[0108] S431: First, the detection of the character target and the object target in the double task, the loss function is generally composed of two parts of classification loss and regression loss, that is, to predict the bbox frame and the target category. The classification loss function uses Focal loss, which is a loss function for handling class imbalance and difficult sample mining. The calculation formula is as follows:

[0109]

[0110] wherein, is the naked result predicted by the model and the probability value after softmax, is the calculation result of the cross-entropy loss, so the is smaller, the prediction is less accurate, and this item will increase, so this item is also used as the coefficient of the difficult sample. The more inaccurate the prediction, the more Focal loss tends to regard the sample as a difficult sample, and the larger the coefficient. The purpose is to make the difficult sample contribute more to the loss and gradient. The is the category weight coefficient. If you have a class- imbalanced dataset, you certainly want to give a high weight to the loss contribution of the class with less quantity. The plays such a role. Therefore, should be a vector, and the length of the vector is equal to the number of categories, which is used to store the weight of each category.

[0111] The regression loss function adopts CIoU Loss to measure the loss of the bbox, which considers the boundary frame length-width loss, increases the similarity between the predicted frame and the real frame, and can converge faster. The calculation formula is as follows:

[0112]

[0113] wherein, is a regularization term introduced from DIoU, which also has the effect of preventing overfitting; alpha is a positive trade-off parameter, and v is used to measure the consistency of the length-width ratio.

[0114] S432: Secondly, the character interaction behavior recognition task in the double task, the recognition content is three parts of two target recognition contents and interaction behavior action recognition content, and then a complete character-verb-object or character-verb-character two types of interaction information is recognized. Each target here uses the Focal loss function in S431. The specific calculation process is the same as above.

[0115] S433: The specific training process adopts a distributed training method, stores data on two or more GPUs for training, builds an optimizer, iteratively trains, adjusts network parameters until the model converges, and uses pre-trained two-branch backbone network weights to speed up the training process.

[0116] S5: Process the human target object features and human interaction behavior features extracted by the model network in the target area, build a deep neural network model for understanding and reasoning the restaurant customer safety monitoring and employee service work content, use customer safety / non-safety behavior action judgment labels and employee work content recognition labels for identification and classification, so as to determine whether there is unsafe and unreasonable behavior in the current picture, and to protect the safety of customers dining and confirm the service content of employees. For the obtained hoi category, the last layer of classification is performed to judge the safety and reasonableness of the behavior.

[0117] It should be understood that the above embodiments are specific descriptions of the preferred embodiments of the present application, which are used to illustrate the technical solutions of the present application, but the present application is not limited to the above embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope defined by the claims of the present application.

Claims

1. A method for recognizing human interaction behavior in a restaurant scene and a security monitoring system, characterized in that, The method comprises: Step one: first determine the video monitoring collection area, the installation position of the shooting device, and confirm whether the installation conditions are met, then frame the collected video pictures and obtain single-frame images with equal time step intervals; Step two: establish a restaurant employee face and human target, customer human, and restaurant common article tool label library, and construct a model capable of identifying human instances and object targets, so as to extract target detection related features; Step three: establish a label library of human body related action behaviors interacting with common articles and tools in the restaurant, and a label library of human interaction behaviors, that is, a human body action label and a human / object / human interaction behavior label, simultaneously evaluate these action behaviors, judge the safety / unsafety and rationality / irrationality of the behaviors, establish a reasonable safety label, and construct a model capable of identifying human interaction behaviors in the restaurant, so as to extract human interaction behavior related features; Step four: then, the collected video pictures are processed in regions, different regions and safety levels are set on the images, a double-task double-flow parallel learning network is constructed, the features of the two task targets are regarded as two modal features to create a double-flow backbone, and the feature information of target recognition and the feature information of human interaction action behaviors are extracted for the two types of human targets in the images, that is, dining customers, working service personnel, and multiple types of object targets in the scene that may interact with people; The construction method of the double-task double-flow parallel learning network of step four is as follows: The double-task double-flow parallel learning network is constructed, the features of the two task targets are regarded as two modal features to create a double-flow backbone, and the feature information of target recognition and the feature information of human interaction action behaviors are extracted for the two types of human targets in the images, that is, dining customers, working service personnel, and multiple types of object targets in the scene that may interact with people, but the same type of encoder is used for extraction: one branch encodes and extracts object and human target detection features, and the other branch encodes and extracts limb pasta features and human interaction action features; After the construction of the double-flow branch backbone, the features output by the same layer of the double-flow branch backbone need to be operated subsequently, the four scale features obtained after the continuous Swin-Transformer block modules in each layer of stage are multiplied, that is, after each completion of down-sampling calculation and window multi-head self-attention calculation, the features from the same picture but different identification tasks are aligned to show the common significant positions; the features of the two branches are aligned in the spatial part and then the channel parts are recalibrated respectively; Step five: processing the human target object features and the human interaction behavior features extracted by the fusion model network in the target area, constructing a deep neural network model for understanding and reasoning the safety monitoring of restaurant customers and the service content of employees, using customer safety / non-safety behavior action judgment labels and employee work content recognition labels for identification and classification, so as to determine whether there is unsafe and unreasonable behavior in the current picture, so as to protect the safety of customers dining and confirm the service content of employees.

Images