A character interaction detection method, system, device and medium based on feature fusion

Abstract

A character interaction detection method, system, device and medium based on feature fusion, the method steps are: constructing a deep neural network; the constructed deep neural network is initialized; the HICO-DET data set and the object frame and its class predicted by the Faster-RCNN network to the HICO-DET data set are merged to generate a new image set; the initialized deep neural network is trained; the new test set is input into the trained deep neural network, the interaction behavior classification prediction is carried out, and the test accuracy of the test set is obtained; the system, device and medium are used to realize the method; the independent features and interaction features are obtained from different feature layers and are fused, the context information of the interaction area can be effectively utilized, the extracted feature information is fully utilized, and the interaction behavior prediction performance is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human interaction detection, and particularly relates to a human interaction detection method, system, device and medium based on feature fusion. BACKGROUND

[0002] Human object interaction (HOI) detection aims to solve the problem of the interaction relationship between people and objects. Unlike common target detection and action recognition tasks, HOI detection requires higher-level visual understanding. HOI detection requires the model to clearly locate the positions of people and objects in an image and correctly predict their interaction behaviors, such as “person riding a horse” and “person driving a car”. By studying HOI detection, the way in which humans recognize the world around them is simulated, and the human interaction behaviors contained in an image are identified, which is a key to automatically understanding the theme of an image and automatically describing the main content of an image. In the prior art, some methods stimulate a multi-modal model corresponding to HOI images and texts through visual and language pre-training or language knowledge injection, but most of the methods focus on specific feature information of people and objects in an image and do not fully utilize context information in a background area.

[0003] A patent application with application number 【202211512924.0】 discloses a human interaction detection method based on YOLOv5. The core problem of the method is regarded as a dual target detection task. The dual detection includes basic target detection of people and objects and special target detection of an interaction area. The interaction area between each person-object pair is focused on, and a complex many-to-many relationship detection is solved. The interaction area is regarded as a special target. The target detection result is obtained, and the interaction detection result is also obtained. The effect of quickly detecting human interaction relationships is achieved.

[0004] For the prior art solution as described above, after predicting the interaction area, a confidence whether an interaction relationship exists, four parameters required for determining an interaction frame, and a score of each interaction action are directly output. The feature information is not used, the context information in the interaction area is not considered in the model, and the model is implemented based on the YOLO algorithm, which limits the prediction performance of the model.

[0005] The patent application with application number 【202210127157.5】 discloses a person interaction relationship recognition method, which provides a person interaction relationship recognition method, including: recognizing a human object and an object object in a target image, obtaining human features of the human object and object features of the object object, and recognizing an action posture of the human object to obtain posture features of the human object; performing fusion processing according to the human features and the posture features to obtain fusion features of the human object; and performing person interaction relationship prediction according to the fusion features and the object features to obtain a person interaction relationship recognition result between the human object and the object object.

[0006] For the prior art solution as described above, the fused features only contain the features of the human object, lack the assistance of global information, do not fully utilize the feature information extracted from the image, ignore the context information between the human body and the object, and increase the model of human key point detection, which leads to an increase in the computational overhead of the entire network and affects the computational performance of the model.

[0007] Based on the above analysis, the prior art has the following technical problems:

[0008] 1. In the prior art, when classifying and predicting action behaviors, only the human features and the object features are usually input into the classification network, and the region feature information in the human-object interaction region is not used, which leads to the loss of part of the information and affects the performance of the model.

[0009] 2. The prior art attempts to improve the performance of the classification method from various angles, but generally involves a relatively complex design, which leads to low computational efficiency and relatively long training time. SUMMARY

[0010] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a person interaction detection method, system, device and medium based on feature fusion, which can effectively utilize the context information of the interaction region, fully utilize the extracted feature information, and has the characteristics of improving the performance of interaction behavior prediction.

[0011] A person interaction detection method based on feature fusion includes the following steps:

[0012] Step 1: Construct a deep neural network;

[0013] Step 2: Initialize the deep neural network constructed in step 1;

[0014] Step 3, merge the HICO-DET dataset with the object frame and its class predicted by the Faster-RCNN network on the HICO-DET dataset to generate a new image set, wherein the new image set includes a new training set and a test set;

[0015] Step 4, train the deep neural network initialized in step 2;

[0016] Step 5, input the new test set obtained in step 3 into the deep neural network trained in step 4 to perform interactive behavior classification prediction and obtain the test accuracy of the test set.

[0017] The step 1 specifically comprises:

[0018] Step 1.1, a six-layer feature extraction network is constructed, and the structure is input layer→1st convolutional layer→1st residual convolutional layer→2nd residual convolutional layer→3rd residual convolutional layer→4th residual convolutional layer;

[0019] Step 1.2, a three-layer feature enhancement network is constructed, and the structure is 1st convolutional layer→2nd convolutional layer→3rd convolutional layer;

[0020] Step 1.3, two three-layer networks with the same structure are constructed as a human independent feature processing network and an object independent feature processing network, and the structure is 1st adaptive pooling layer→1st full connection layer→2nd full connection layer;

[0021] Step 1.4, a three-layer interactive feature processing network is constructed, and the structure is 1st adaptive pooling layer→1st full connection layer→2nd full connection layer;

[0022] Step 1.5, a two-layer interactive behavior classification network is constructed, and the structure is 1st full connection layer→2nd full connection layer.

[0023] The specific method of the step 2 is:

[0024] Step 2.1, randomly select a number in the range of (0, 1), and use the selected number as the initial weight of each node in the deep neural network;

[0025] Step 2.2, use the initial weight of each node as the weight of each node in the deep neural network in the first iteration process.

[0026] The HICO-DET data set in step 3 includes a training set and a test set, and the training set and the test set both contain real person boxes and object boxes and classification labels of the person boxes and the object boxes, the images in the training set and the test set in the HICO-DET data set are input into the Faster-RCNN network to predict the object boxes and the categories of the object boxes, and the predicted object boxes and the categories of the object boxes are combined with the HICO-DET data set to generate new training sets and test sets.

[0027] The specific method of step 4 is:

[0028] Step 4.1, randomly take R images from the new training set generated in step 3, wherein R is a data batch processing size, and input the R images into the feature extraction network in step 1.1 to obtain backbone features;

[0029] Step 4.2, according to the size of the R images randomly taken in step 4.1 and the size of the backbone features extracted in the feature extraction network, scale the person boxes and the object boxes;

[0030] Step 4.3, obtain independent features and interaction features through the feature extraction network in step 1.1 and the feature enhancement network in step 1.2; specifically:

[0031] The output of the third residual convolutional layer of the feature extraction network in step 1.1 is input into the feature enhancement network in step 1.2 to obtain enhanced backbone features, the independent features are cut out from the enhanced backbone features according to the size of the scaled person boxes and object boxes in step 4.2, and the interaction features are cut out from the output features of the fourth residual convolutional layer of the feature extraction network in step 1.1 according to the box information of the interaction region;

[0032] Step 4.4, perform feature fusion operation on the independent features and the interaction features cut out in step 4.3;

[0033] Step 4.5, input the fused features in step 4.4 into the interaction behavior classification network constructed in step 1.5 to perform category prediction, and finally the interaction behavior classification network outputs the predicted classification labels of the interaction regions in the R images;

[0034] Step 4.6, calculate the Focal loss value of the correct classification of each sample in the R images in step 4.1 according to the predicted classification labels obtained in step 4.5;

[0035] Step 4.7, use the Focal loss value calculated in step 4.6 to calculate the partial derivative of the weight value of each node in the deep neural network initialized in step 2, to obtain the gradient value of the weight value of each node in the deep neural network;

[0036] Step 4.8, update all parameters in the deep neural network initialized in step 2 with the gradient value of each node calculated in step 4.7;

[0037] Step 4.9, input the new test set generated in step 3 into the deep neural network updated in step 4.8 in turn, output the predicted classification label corresponding to each image, and calculate the accuracy of the deep neural network on the test set;

[0038] Step 4.10, determine whether the accuracy in step 4.9 is still rising, if not, get the trained deep neural network, otherwise, execute step 4.1.

[0039] In step 1.2, the feature enhancement network parameters are set as follows: the total number of feature maps of the first convolutional layer is set to 1024, the size of the convolution kernel is set to 1x1 pixels, and the step size is set to 1 pixel; the total number of feature maps of the second convolutional layer is set to 1024, the size of the convolution kernel is set to 3x3 pixels, and the step size is set to 2 pixels; the total number of feature maps of the third convolutional layer is set to 1024, the size of the convolution kernel is set to 1x1 pixels, and the step size is set to 1 pixel; the output of the second convolutional layer is subjected to L2 regularization, and the output of the third convolutional layer is subjected to matrix multiplication as the final output.

[0040] The specific method of step 4.4 is as follows:

[0041] The independent features and interaction features extracted in step 4.3 are respectively input into the independent feature processing network of step 1.3 and the interaction feature processing network of step 1.4 for dimension reduction operation, and then the independent features and interaction features are fused according to the following formula:

[0042]

[0043] Where, f l,i represents the independent features of people and objects, i∈{h,o} represents whether the feature belongs to people or objects, f g represents the interaction features, f l,i ·f g represents matrix dot product and summation, the corresponding position elements in the two features are multiplied, |f g | 2 is the L2 regularization of f g , and then the component orthogonal to f g is obtained according to the following formula:

[0044] f l_f,i =f l,i -f l_m,i

[0045] Stack the finally obtained independent features and the interaction features.

[0046] The application further provides a character interaction detection system based on feature fusion, comprising:

[0047] A model construction module is configured to construct a deep neural network and generate the entire deep neural network.

[0048] A data acquisition module is configured to merge a HICO-DET dataset with object frames and their categories predicted by a Faster-RCNN network for the HICO-DET dataset, and generate a new image set, wherein the new image set comprises a new training set and a test set.

[0049] A feature enhancement module is configured to input the output of the third residual convolutional layer of the feature extraction network into the feature enhancement network to obtain enhanced backbone features.

[0050] A feature interception module is configured to obtain independent features and interaction features, intercept the independent features from the enhanced backbone features according to the sizes of the character frames and the object frames, and intercept the interaction features from the output features of the fourth residual convolutional layer of the feature extraction network according to the frame information of the interaction regions.

[0051] A feature fusion module is configured to fuse the independent features and the interaction features to obtain features used for interaction behavior classification prediction.

[0052] A prediction output module is configured to perform interaction behavior classification prediction on images, and output all obtained interaction behavior predictions.

[0053] The application further provides a character interaction detection device based on feature fusion, comprising:

[0054] A memory is configured to store a computer program of the character interaction detection method based on feature fusion, and the memory is a computer-readable device.

[0055] A processor is configured to execute the computer program to realize the character interaction detection method based on feature fusion.

[0056] The application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executable by a processor to realize the character interaction detection method based on feature fusion.

[0057] Compared with the prior art, the application has the following beneficial effects:

[0058] 1、The application not only uses independent features, but also uses interactive features, can consider the context information in the interactive background area while using the specific features of the characters and objects, overcomes the insufficient use of context information in the background area of the prior art, thereby expanding the integrity of the features, enabling the deep neural network to effectively learn the semantic features between the characters and objects, and improving the classification performance of the application.

[0059] 2、The application fuses independent features and interactive features by using the proposed feature fusion algorithm, and transmits the fused independent features and interactive features into the subsequent network, enhances the integrity of the final features, enables the network to consider whether the independent features really have interactive behavior, uses the fused feature information in the classification stage, can extract more effective information, makes the model converge faster, and the training time is much smaller than the model based on the Transformer, and the performance is good after about ten iterations.

[0060] 3、The application uses a feature enhancement module when processing independent features, can effectively reduce the dimension of independent features, reduce the calculation overhead, and make the deep neural network pay more attention to important areas in each independent feature, thereby enhancing the robustness of the independent features.

[0061] 4、The application extracts feature information from different layers of the backbone network, extracts independent features from the second-to-last layer of the backbone network, and shallow network pays more attention to image features, extracts interactive features from the last layer of the backbone network, and deep network pays more attention to semantic features, and the fused features can obtain features that take into account both semantic features and image features. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flowchart of the application.

[0063] Figure 2 is a network model diagram of the application.

[0064] Figure 3 is a model diagram of the feature enhancement network of the application.

[0065] Figure 4 is a flowchart of the training neural network stage of the application.

[0066] Figure 5 is a model diagram of the feature fusion module of the application. DETAILED DESCRIPTION

[0067] The application will be further described below with reference to the accompanying drawings.

[0068] The idea for achieving the object of the present application is: first, the image data set is transmitted into the Faster-RCNN network to obtain the prediction of the human frame and the object frame for the subsequent input preparation. Subsequently, a deep neural network capable of extracting independent features and interactive features and fusing them is constructed, the image data set and the predicted human frame and object frame are used for training, the independent features are obtained according to the human frame and the object frame, the interactive features are obtained according to the interactive region of the human and the object, then the independent features and the interactive features are fused according to the fusion algorithm and are transmitted into the subsequent classification network for classification training, and a trained deep learning network is obtained. In the prediction, the image is directly transmitted into the trained network model, and finally the model frames the human and the object with interactive behaviors and lists the interactive behaviors between the two.

[0069] Reference will now be made to the drawings, in which Figure 1 The implementation steps of the present application are described in detail.

[0070] A human interaction detection method based on feature fusion, comprising the following steps:

[0071] Step 1, a deep neural network is constructed, which is composed of a feature extraction network, an independent feature processing network, an interactive feature processing network and an interactive behavior classification network;

[0072] Reference will now be made to the drawings, in which Figure 2 The deep neural network constructed by the present application is further described in detail.

[0073] Step 1.1, a six-layer feature extraction network is constructed for extracting the backbone features of the image, and the structure is input layer→1st convolutional layer→1st residual convolutional layer→2nd residual convolutional layer→3rd residual convolutional layer→4th residual convolutional layer in turn;

[0074] In the step 1.1, the parameters of each layer of the feature extraction network are set as follows: the total number of feature mapping diagrams of the input layer is set to 3; the total number of feature mapping diagrams of the 1st convolutional layer is set to 64, the convolution kernel size is set to 7x7 pixels, and the step size is set to 2 pixels; the total number of feature mapping diagrams of the 1st residual convolutional layer is set to 256, the convolution kernel size is set to 3x3 pixels, and the step size is set to 2 pixels; the total number of feature mapping diagrams of the 2nd residual convolutional layer is set to 512, the convolution kernel size is set to 3x3 pixels, and the step size is set to 2 pixels; the total number of feature mapping diagrams of the 3rd residual convolutional layer is set to 1024, the convolution kernel size is set to 3x3 pixels, and the step size is set to 2 pixels; the total number of feature mapping diagrams of the 4th residual convolutional layer is set to 2048, the convolution kernel size is set to 3x3 pixels, and the step size is set to 2 pixels.

[0075] Step 1.2, a three-layer feature enhancement network is constructed, and the structure is: the first convolutional layer→the second convolutional layer→the third convolutional layer;

[0076] Referring to the drawings Figure 3 , the feature enhancement network is further described.

[0077] In step 1.2, the parameters of the feature enhancement network are set as follows: the total number of feature maps of the first convolutional layer is set to 1024, the size of the convolution kernel is set to 1×1 pixels, and the step size is set to 1 pixel; the total number of feature maps of the second convolutional layer is set to 1024, the size of the convolution kernel is set to 3×3 pixels, and the step size is set to 2 pixels; the total number of feature maps of the third convolutional layer is set to 1024, the size of the convolution kernel is set to 1×1 pixels, and the step size is set to 1 pixel; the output of the second convolutional layer is subjected to L2 regularization, and the output of the third convolutional layer is subjected to matrix multiplication as the final output.

[0078] Step 1.3, two three-layer networks with the same structure are constructed as human independent feature processing network and object independent feature processing network, and the structure is: the first adaptive pooling layer→the first fully connected layer→the second fully connected layer;

[0079] In step 1.3, the parameters of the independent feature processing network are set as follows: the output feature map size of the adaptive pooling layer is set to 7×7; the input and output node numbers of the first fully connected layer are set to the product of the width, height and total number of feature maps of the pooling layer output feature and 1024, respectively; the input and output node numbers of the second fully connected layer are set to 1024 and 1024, respectively.

[0080] Step 1.4, a three-layer interactive feature processing network is constructed, and the structure is: the first adaptive pooling layer→the first fully connected layer→the second fully connected layer;

[0081] In step 1.4, the parameters of the interactive feature processing network are set as follows: the output feature map size of the adaptive pooling layer is set to 7×7; the input and output node numbers of the first fully connected layer are set to the product of the width, height and total number of feature maps of the pooling layer output feature and 2048, respectively; the input and output node numbers of the second fully connected layer are set to 1024 and 1024, respectively.

[0082] Step 1.5, a two-layer interactive behavior classification network is constructed, and the structure is: the first fully connected layer→the second fully connected layer.

[0083] In step 1.5, the interactive behavior classification network parameters are set as follows: the input and output node numbers of the first full connection layer are set to 3072 and 1024 respectively, and the input and output node numbers of the second full connection layer are set to 1024 and the number of categories respectively.

[0084] Step 2, initialize the deep neural network constructed in step 1;

[0085] The specific method of step 2 is:

[0086] Step 2.1, randomly select a number in the range of (0, 1), and use the selected number as the initial weight of each node in the deep neural network;

[0087] Step 2.2, use the initial weight of each node as the weight of each node in the deep neural network in the first iteration process.

[0088] Step 3, merge the HICO-DET dataset and the object frame and its category predicted by the Faster-RCNN network for the HICO-DET dataset to generate a new image set, the new image set including a new training set and a test set;

[0089] The HICO-DET dataset includes a training set and a test set, and both the training set and the test set contain real person frames and object frames and the classification labels of the person frames and the object frames. The images in the training set and the test set in the HICO-DET dataset are predicted by the Faster-RCNN network to obtain the object frame and its category, and the predicted object frame and its category are merged with the HICO-DET dataset to generate a new training set and a test set. The new training set and the test set are used as input data of the deep neural network.

[0090] Step 4, train the deep neural network initialized in step 2.

[0091] Refer to the accompanying Figure 4 Further detailed description is made to the training process of the deep neural network of the application.

[0092] Step 4.1, randomly take out 4 images from the new training set generated in step 3, input the randomly taken out 4 images into the feature extraction network in step 1.1 to obtain the backbone feature, and use the deep neural network initialized in step 2 as the deep neural network for subsequent weight updating;

[0093] Step 4.2, according to the size of the R images randomly taken out in step 4.1 and the size of the backbone feature extracted from the feature extraction network, scale the person frame and the object frame;

[0094] According to the information of the predicted human frame and object frame in each of the four images, the frame information of the interaction region of the human and the object is obtained according to the following formula:

[0095]

[0096] wherein, h i represents the information of the i-th human frame, o j represents the information of the j-th object frame, x1, y1 respectively represent the coordinates of the upper left corner of the frame, and x2, y2 represent the coordinates of the lower right corner of the frame.

[0097] According to the size of each of the four images and the size of the backbone feature, the size of the frame is proportionally reduced according to the following formula:

[0098]

[0099]

[0100] wherein, image i represents the i-th image, feature i represents the backbone feature corresponding to the i-th image, ori_boses i represents the size of the initial frame in the i-th image, X ∈ {human, object, interation} respectively represents that the frame information belongs to human, object and interaction region, the subscripts x, y of the image and the backbone feature respectively represent the width and the height, and the final new proportionally reduced frame information is generated, and then the backbone network is cropped according to the frame information.

[0101] Step 4.3, independent features and interaction features are obtained through the feature extraction network of step 1.1 and the feature enhancement network of step 1.2; specifically:

[0102] The output of the third residual convolutional layer of the feature extraction network of step 1.1 is transmitted into the feature enhancement network of step 1.2 to obtain enhanced backbone features, the independent features are cut out from the enhanced backbone features according to the size of the scaled human frame and object frame of step 4.2, and the interaction features are cut out from the output features of the fourth residual convolutional layer of the feature extraction network of step 1.1 according to the frame information of the interaction region;

[0103] Step 4.4, the independent features and the interaction features cut out in step 4.3 are subjected to feature fusion operation to obtain features used for interaction behavior classification prediction in step 1.5.

[0104] Referring to the accompanying drawings Figure 5 , the feature fusion algorithm is further described.

[0105] The independent features and the interaction features intercepted in step 4.3 are respectively input into the step 1.3 independent feature processing network and the step 1.4 interaction feature processing network for dimension reduction operation, and then the independent features and the interaction features are fused according to the following formula:

[0106]

[0107] wherein f l,i represents the independent features of the human and the object, i represents whether the feature belongs to the human or the object, f g represents the interaction features, f l,i ·f g represents the matrix point multiplication and then summation, the corresponding position elements in the two features are multiplied, |f g | 2 is the L2 norm of f g , and then the component orthogonal to f g is obtained according to the following formula:

[0108] fl _f,i =f l,i -fl _m,i

[0109] The finally obtained independent features and the interaction features are stacked;

[0110] Step 4.5, the fused features in step 4.4 are input into the interaction behavior classification network constructed in step 1.5 for category prediction, and the final interaction behavior classification network outputs the predicted classification label of the interaction region in the four images;

[0111] Step 4.6, according to the predicted classification label obtained in step 4.5, the Focal loss value of the correct classification of each sample of the four images in step 4.1 is calculated according to the following formula:

[0112]

[0113] wherein FL is the total Focal loss value, N represents the total sum of the training samples, ∑ represents the summation operation, i represents the sample serial number, is the predicted classification label of the interaction region of the i-th in step 4.5, y i ∈{0,1} is the true binary label of the i-th interaction region in step 4.5, and γ∈R + is a hyperparameter;

[0114] Step 4.7, the Focal loss value calculated in step 4.6 is used to derive the gradient value of the weight of each node in the deep neural network initialized in step 2.

[0115] Step 4.8, update all the parameters in the deep neural network initialized in step 2 using the gradient values calculated in step 4.7 with backpropagation algorithm.

[0116] The backpropagation algorithm is as follows:

[0117] Update the weight of each node in the deep neural network initialized in step 2 as follows:

[0118]

[0119] wherein, represents the updated weight of the cth node in the deep neural network initialized in step 2, g c represents the weight of the cth node in the deep neural network initialized in step 2 before updating, β represents the learning rate of the deep neural network initialized in step 2, represents the gradient value of the weight of the cth node in the deep neural network initialized in step 2 calculated in step 4.7.

[0120] Step 4.9, input the new test set generated in step 3 into the deep neural network updated in step 4.8 in turn, output the predicted classification label corresponding to each image, and calculate the accuracy of the deep neural network on the test set using the test accuracy calculation formula.

[0121] The calculation formula is as follows:

[0122] First, calculate the accuracy AP of each class in the new test set generated in step 3:

[0123]

[0124]

[0125] wherein TP k represents the number of the kth class classification labels in the new test set generated in step 3 that are correctly predicted by the predicted classification label output in this step, FP k represents the number of the kth class classification labels in the new test set generated in step 3 that are incorrectly predicted by the predicted classification label output in this step, FN k represents the number of the kth class real labels in the new test set generated in step 3 that are not predicted, and Precision k and Recall k are both sequences with the size of the number of predicted classification labels of the kth class, and each Precision corresponds to a Recall, because each time a predicted classification label is judged as TP / FP / FN, Precision and Recall are calculated.

[0126] Then, the recall is divided into {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} by using 11-point sampling method. The precision of the recall in the interval [0, 1] is obtained according to the following formula, and 11 points of accuracy P are obtained:

[0127]

[0128] wherein i is in {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, and respectively represent the jth recall and precision in the kth class label of the new test set generated in step 3.

[0129]

[0130] wherein, represents the ith accuracy in the kth predicted classification label of the new test set generated in step 3, i is in {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}.

[0131] Finally, the accuracy of all classes in the new test set generated in step 3 is calculated and averaged to obtain mAP:

[0132]

[0133] wherein m represents the total number of classes in the new test set generated in step 3, and AP i represents the accuracy of the ith class.

[0134] Step 4.10, determine whether the accuracy in step 4.9 is still increasing, if not, obtain the trained deep neural network, otherwise, execute step 4.1.

[0135] Step 5, input the new test set obtained in step 3 into the deep neural network trained in step 4.10 to perform interactive behavior classification prediction, and obtain the test accuracy of the test set.

[0136] The application also provides a character interaction detection system based on feature fusion, comprising:

[0137] A model construction module is used to realize the construction of the deep neural network in step 1 and generate the entire deep neural network.

[0138] The data acquisition module is used for merging the HICO-DET dataset with the object frame and the class thereof predicted by the Faster-RCNN network in step 3, and generating a new image set, wherein the new image set comprises a new training set and a test set.

[0139] The feature enhancement module is used for transmitting the output of the third residual convolutional layer of the feature extraction network into the feature enhancement network in step 1.2 to obtain the enhanced backbone feature in step 4.

[0140] The feature extraction module is used for obtaining the independent feature and the interaction feature in step 4.3, wherein the independent feature is obtained by cutting the enhanced backbone feature according to the size of the person frame and the object frame, and the interaction feature is obtained by cutting the output feature of the fourth residual convolutional layer of the feature extraction network in step 1.1 according to the frame information of the interaction region.

[0141] The feature fusion module is used for fusing the independent feature and the interaction feature to obtain the feature used for the interaction behavior classification prediction in step 1.5 in step 4.4.

[0142] The prediction output module is used for predicting the interaction behavior classification of the image and outputting all the interaction behavior predictions in step 5.

[0143] The application further provides a person interaction detection device based on feature fusion, which comprises:

[0144] The memory is used for storing the computer program of the person interaction detection method based on feature fusion, and the memory is a computer-readable device.

[0145] The processor is used for executing the computer program to realize the person interaction detection method based on feature fusion.

[0146] The application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can realize the person interaction detection method based on feature fusion when the computer program is executed by a processor.

[0147] The effects of the application are further illustrated by the following simulation experiment.

[0148] 1. Simulation experiment conditions:

[0149] The hardware platform of the simulation experiment of the application is: Intel(R) Core(TM) i7-8700K CPU, 3.70GHz*12, NVIDIA GeForce GTX 3090Ti, and the memory is 64G.

[0150] The software platform of the simulation experiment of the application is: pytorch.

[0151] 2. Simulation content and result analysis:

[0152] The simulation experiment of the application uses a labeled human interaction detection dataset to train a deep neural network, obtains a trained deep neural network, and inputs an original image into the deep learning network for category judgment.

[0153] The human interaction detection dataset used in the simulation experiment of the application is the HICO-DET dataset, which has a total of 47179 interaction images, of which the training set has 37633 images and the test set has 9546 images. The training set is used to iteratively train the deep neural network for 10 times to obtain a trained deep neural network. Then, the test sample set is used to test the trained deep neural network to obtain the recognition accuracy of the application.

[0154] The recognition accuracy of the method of the application and the prior art method under the same dataset is compared, and the recognition accuracy results of several methods after training are shown in Table 1:

[0155] Table 1 Comparison table of human interaction detection classification results

[0156]

[0157] As can be seen from Table 1, the recognition accuracy of the human interaction detection classification method based on the feature fusion neural network proposed in the application is improved by 0.92% compared with the classification method of the prior art, which shows that the method of fusing independent features and interaction features in the application can improve the classification accuracy of human and object interaction behaviors in images, and has important practical significance. Among them, the names of the four comparison methods are as follows:

[0158] SCG: [Zhang F Z, Campbell D, Gould S. Spatially Conditioned Graphs for Detecting Human-Object Interactions [C] / / International Conference on Computer Vision. 2021.];

[0159] QPIC: [Tamura M, Ohashi H, Yoshinaga T. QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information:, 10.48550 / arXiv.2103.05399 [P]. 2021.];

[0161] SSRT: [Zhong X, Ding C, Li Z, et al. Towards Hard-Positive Query Mining for DETR-Based Human-Object Interaction Detection [C] / / European Conference on Computer Vision. Springer, Cham, 2022.];

[0162] UTP: [Zhang F Z, Campbell D, Gould S. Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer [J]. 2021.].

[0163] As shown in the flowchart of the present application. It can be observed that the present application is divided into five steps, constructing a deep neural network, initializing the deep neural network, generating an image set, training the deep neural network and classifying the new test set. Figure 1 As shown in the network model of the present application, it can be observed that the model is divided into five parts, backbone feature extraction part, independent feature branch processing part, interaction feature branch processing part, feature fusion part and classification part, wherein P represents the AVGPooling operation.

[0164] Figure 2 As shown in the network model of the present application, it can be observed that the model is divided into five parts, backbone feature extraction part, independent feature branch processing part, interaction feature branch processing part, feature fusion part and classification part, wherein P represents the AVGPooling operation.

[0165] As shown in the network model of the present application, it can be observed that the model is divided into five parts, backbone feature extraction part, independent feature branch processing part, interaction feature branch processing part, feature fusion part and classification part, wherein P represents the AVGPooling operation. Figure 3 ​The model diagram of the feature enhancement network of the application is shown, and it can be observed that the part is mainly composed of a convolution layer, an activation layer and a BN layer, in addition, L2 regularization is performed after the output of the second convolution layer, and then the matrix multiplication operation is performed with the output of the third convolution layer to obtain the final output; wherein C, c, k represent the convolution layer with c as the channel number and kxk as the convolution kernel, R represents the ReLU activation function, B represents the BN, S represents the Softplus, and the last * represents the matrix point multiplication.

[0166] As shown in Figure 4 The flow chart of the training neural network stage of the application is shown. It can be observed that the network training stage of the application is divided into ten steps, obtaining the backbone feature, obtaining the interaction region and reducing it by equal ratio, obtaining the independent feature and the interaction feature, feature fusion, class prediction, loss calculation, judging whether the test set accuracy is still rising, still rising to obtain the network node gradient and update the network node weight; no longer rising to obtain the trained network model.

[0167] As shown in Figure 5 The model diagram of the feature fusion module of the application is shown, and it can be observed that the independent feature and the interaction feature are fused and stacked in this part; wherein f l,i represents the local feature of the person and the object, i represents whether the feature belongs to the person or the object, f g represents the global feature, f l,i ·f g represents the matrix point multiplication and summation, the corresponding position elements in the two features are multiplied, |f g | 2 is the L2 regularization of f g , - represents the matrix subtraction operation, and C represents the aggregation operation.

[0168] Application prospect of the application:

[0169] Firstly, the person interaction detection is a problem with very important practical significance, is the key to realize automatic understanding of image theme and automatic description of image main content, can be applied to the field of video monitoring, and classifies and supervises the human behavior.

[0170] The present application starts from a key problem of human interaction detection, that is, the feature information extracted from the backbone network is not fully utilized in the model training process, which is an innovative and practical step in the current problem and method prospect. The innovation lies in the fact that the present application creatively utilizes the feature information in the human and object interaction area, and simultaneously extracts independent features and interaction features from different layers of the backbone feature extraction network for fusion, which is not used in previous methods. The practical application lies in the fact that the method proposed by the present application can effectively predict the objects having interaction behavior with humans and the interaction behavior therebetween. The method does not have too complex design, so it can easily deploy training preparation learning, and has high calculation efficiency and less training time, so it can quickly train and learn, and then be put into practical application.

Claims

1. A method for human interaction detection based on feature fusion, characterized in that: The method comprises the following steps: Step 1, constructing a deep neural network; Specifically comprising: Step 1.1, constructing a six-layer feature extraction network, the structure of which is input layer→1st convolutional layer→1st residual convolutional layer→2nd residual convolutional layer→3rd residual convolutional layer→4th residual convolutional layer in turn; Step 1.2, constructing a three-layer feature enhancement network, the structure of which is 1st convolutional layer→2nd convolutional layer→3rd convolutional layer in turn; Step 1.3, constructing two three-layer networks with the same structure as the human independent feature processing network and the object independent feature processing network, the structure of which is 1st adaptive pooling layer→1st full connection layer→2nd full connection layer in turn; Step 1.4, constructing a three-layer interactive feature processing network, the structure of which is 1st adaptive pooling layer→1st full connection layer→2nd full connection layer in turn; Step 1.5, constructing a two-layer interactive behavior classification network, the structure of which is 1st full connection layer→2nd full connection layer in turn; Step 2, initializing the deep neural network constructed in step 1; Step 3, merging the HICO-DET dataset with the object frame and its category predicted by the Faster-RCNN network on the HICO-DET dataset to generate a new image set, wherein the new image set comprises a new training set and a test set; Step 4, training the deep neural network initialized in step 2; the specific method is: Step 4.1, randomly taking out R images from the new training set generated in step 3, wherein R is the data batch processing size, and inputting the R images into the feature extraction network in step 1.1 to obtain the backbone feature; Step 4.2, according to the size of the R images randomly taken out in step 4.1 and the size of the backbone feature extracted from the feature extraction network, scaling the human frame and the object frame; Step 4.3, obtaining the independent feature and the interactive feature through the feature extraction network in step 1.1 and the feature enhancement network in step 1.2; specifically: The output of the 3rd residual convolutional layer of the feature extraction network in step 1.1 is input into the feature enhancement network in step 1.2 to obtain the enhanced backbone feature, the independent feature is cut out from the enhanced backbone feature according to the size of the human frame and the object frame scaled in step 4.2, and the interactive feature is cut out from the output feature of the 4th residual convolutional layer of the feature extraction network in step 1.1 according to the frame information of the interactive region; Step 4.4, performing feature fusion operation on the independent feature and the interactive feature cut out in step 4.3; the specific method is: The independent feature and the interactive feature cut out in step 4.3 are respectively input into the independent feature processing network in step 1.3 and the interactive feature processing network in step 1.4 for dimension reduction operation, and then the independent feature and the interactive feature are fused according to the following formula: where f l,i represents independent features of people and objects, i ∈ {h, o} indicates whether the feature belongs to people or objects, f g represents interaction features, f l,i · f g represents matrix point multiplication and summation, the corresponding position elements of the two features are multiplied, |f g | 2 is the L2 regularization of f g , and then the component orthogonal to f g is obtained according to the following formula: f l_f,i = f l,i -f l_m,i Stacking the finally obtained independent feature and interactive feature; Step 4.5, inputting the feature fused in step 4.4 into the interactive behavior classification network constructed in step 1.5 for category prediction, and finally the interactive behavior classification network outputs the prediction classification label of the interactive region in the R images; Step 4.6, calculate the Focal loss value of each sample in the R images in step 4.1 correctly classified according to the predicted classification label obtained in step 4.5; Step 4.7, the Focal loss value calculated in step 4.6 is used to calculate the partial derivative of the weight value of each node in the deep neural network initialized in step 2, and the gradient value of each node weight in the deep neural network is obtained; Step 4.8, update all parameters in the deep neural network initialized in step 2 with the gradient value of each node calculated in step 4.7; Step 4.9, input the new test set generated in step 3 into the deep neural network updated in step 4.8 in turn, output the corresponding predicted classification label of each image, and calculate the accuracy of the deep neural network on the test set; Step 4.10, judge whether the accuracy in step 4.9 is still rising, if not, the trained deep neural network is obtained, otherwise, step 4.1 is executed; Step 5, input the new test set obtained in step 3 into the deep neural network trained in step 4 to perform interactive behavior classification prediction, and obtain the test accuracy of the test set. 2.The human interaction detection method based on feature fusion according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1, randomly select a number in the range of (0, 1), and use the selected number as the initial weight value of each node in the deep neural network; Step 2.2, the initial weight value of each node is used as the weight value of each node in the deep neural network in the first iteration process. 3.The human interaction detection method based on feature fusion according to claim 1, characterized in that: The HICO-DET data set in step 3 includes a training set and a test set, and the training set and the test set both contain real person frames and object frames and the classification labels of the person frames and the object frames. The images in the HICO-DET data set are combined with the measured object frames and their categories to generate a new training set and a test set.

4. The method of claim 1, wherein the method is based on feature fusion. In step 1.2, the feature enhancement network parameters are set as follows: the total number of feature maps of the first convolutional layer is set to 1024, the convolution kernel size is set to 1×1 pixel, and the step size is set to 1 pixel; the total number of feature maps of the second convolutional layer is set to 1024, the convolution kernel size is set to 3×3 pixels, and the step size is set to 2 pixels; the total number of feature maps of the third convolutional layer is set to 1024, the convolution kernel size is set to 1×1 pixel, and the step size is set to 1 pixel; the output of the second convolutional layer is subjected to L2 regularization, and the output of the third convolutional layer is subjected to matrix multiplication operation as the final output.

5. A feature fusion based human interaction detection system based on the method of claim 1, characterized in that: It comprises: A model construction module for constructing a deep neural network to generate the entire deep neural network; A data acquisition module for merging the HICO-DET data set and the object frames and their categories predicted by the Faster-RCNN network on the HICO-DET data set to generate a new image set, wherein the new image set includes a new training set and a test set; A feature enhancement module for inputting the output of the third residual convolutional layer of the feature extraction network into the feature enhancement network to obtain enhanced backbone features; A feature enhancement module for inputting the output of the third residual convolutional layer of the feature extraction network into the feature enhancement network to obtain enhanced backbone features; The feature extraction module is configured to acquire the independent feature and the interaction feature, the independent feature is extracted from the enhanced backbone feature according to the size of the person frame and the object frame, and the interaction feature is extracted from the output feature of the fourth residual convolution layer of the feature extraction network according to the frame information of the interaction region. The feature fusion module is configured to fuse the independent feature and the interaction feature to obtain the feature used for interaction behavior classification prediction. The prediction output module is configured to perform interaction behavior classification prediction on the image, and output all the obtained interaction behavior predictions.

6. A person interaction detection device based on feature fusion, characterized in that: The computer program for the character interaction detection method based on feature fusion according to any one of claims 1-4 is stored in the memory, and the computer program is readable by the computer. The processor is configured to execute the computer program to realize the character interaction detection method based on feature fusion according to any one of claims 1-4. The computer program is stored in the computer readable storage medium, and the computer program is executable by the processor to realize the character interaction detection method based on feature fusion according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: ​

Images